12:31:36 <jzb> #startmeeting flock2016
12:31:36 <zodbot> Meeting started Tue Aug  2 12:31:36 2016 UTC.  The chair is jzb. Information about MeetBot at http://wiki.debian.org/MeetBot.
12:31:36 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
12:31:36 <zodbot> The meeting name has been set to 'flock2016'
12:31:51 <jzb> Dan Walsh doing talk on Containers in Production
12:32:05 <jzb> put out paper internally, but will be exteral, about containers in production
12:32:33 <jzb> one thing - talk about devops all the time. New model putting out packages, release cycles, at this point, Docker is concentrated almost exclusively on the developer side.
12:32:51 <jzb> really concentrating on the developer. Trying to look from the operators point of view, what are operators looking for?
12:33:04 <jzb> Most container work concentrated on developer, docker-build docker commit
12:33:16 <jzb> standardization ,new tools can arise
12:33:27 <jzb> starting to see other groups starting to build container images using other tools
12:33:31 <jzb> good to have innovation
12:33:34 <jzb> ansible has playbooks
12:33:42 <jzb> openshift has a thing called s2i (Source to Image)
12:33:58 <jzb> talk about how Fedora is doing layered builds, another project going on for a while...
12:34:16 <jzb> as long as output is the same, we don't have to ship a Docker fully instrumented for building containers in production.
12:34:20 <jzb> copy on write filesystem
12:34:29 <jzb> when doing docker builds and commits, what is that?
12:34:44 <jzb> taking what I had before, COW filesystem
12:35:05 <jzb> when I'm running in production, I don't want application to be able to modify itself. we want to move towards a state where we're not using a COW filesystem.
12:35:11 <jzb> COWs have some problems
12:35:22 <jzb> DeviceMapper, btrfs break shared memor
12:35:25 <jzb> memory
12:35:49 <jzb> block based devices, problem with filesystem, kernel figures out that shared memory is the same.
12:36:01 <jzb> kernel looks @ filesystem, to see block device is the same
12:36:15 <jzb> as soon as I use DM or btrfs, kernel gets confused, both systems are using same lower-level image
12:36:41 <jzb> 10 containers ... with JRE, on a standard filesystem, you load 1 into memory, with cow you get 10
12:36:49 <jzb> overlayfs - doesn't have shared memory
12:36:59 <jzb> working very hard to fix it, SELinux support isn't there
12:37:04 <jzb> SElinux coming soon
12:37:39 <jzb> OverlayFS - not posix compliant filesystem
12:37:52 <jzb> problems based on assumptions with it being posix compliant
12:38:18 <jzb> open file for write, and open for read only, can wind up with two separate files.
12:38:23 <jzb> not the same file twice
12:38:55 <jzb> write performance suffers at larger scales with COW
12:39:04 <jzb> in production most images should never be modified
12:39:14 <jzb> don't want applications able to write to /usr when in production
12:39:30 <jzb> security much better if only can write to certain directories, COW do not support network storage
12:39:34 <jzb> doesn't work with NFS
12:39:38 <jzb> lose ability to share images
12:39:52 <jzb> each container server downloads gigabytes of images
12:41:26 <jzb> shared images == instant updates - all containers running the same code
12:41:33 <jzb> get rid of cow filesystems?
12:42:01 <jzb> in production we want to, but we can't
12:42:05 <jzb> we have pet filesystems
12:42:11 <jzb> correction: Pet containers
12:42:29 <jzb> atomic/openshift registry, taking docker registry service + additional features
12:42:40 <jzb> when someone does docker push to registry
12:42:45 <jzb> watches for messages about push
12:42:50 <jzb> application explodes onto ostree
12:43:03 <jzb> explodes filesystem to ostree filesystem
12:43:08 <jzb> first ability to do an atomic scan
12:43:16 <jzb> this looks for vulnerabilities
12:43:41 <jzb> approve or block for sharing via docker pull
12:45:55 <jzb> [video playing]
12:52:59 <jzb> system containers
12:53:19 <jzb> on an atomic host, software must be shipped as a container image
12:53:31 <jzb> how do you handle when software is installed as container?
12:53:41 <jzb> kubernetes requires etcd and flanneld.
12:53:50 <jzb> they have to be up before k8s, and docker needs all of those.
12:53:59 <jzb> these containers can be run with read/only images
12:54:12 <jzb> docker has problems with container priority
12:54:34 <jzb> but systemd can give priority to other things, but can't install etcd and flanneld
12:54:37 <jzb> and need docker
12:54:46 <jzb> so - came up with concept of system containers
12:54:59 <jzb> atomic command enhanced to handle system containers
12:55:15 <jzb> uses skopeo to pull image from registry
12:55:21 <jzb> use ostree to store image layers on disk
12:55:31 <jzb> create systemd unit file and uses runc to run container
12:55:47 <jzb> runc is a standardized part of open container initiative
12:55:57 <jzb> going to use runc inside systemd unit files
12:56:04 <jzb> atomic pull --oci etcd
12:56:44 <jzb> atomic install --oci etcd
12:56:57 <jzb> systemd enable etc
12:57:01 <jzb> ...
12:57:11 <jzb> using runc, systemd, ostree for backing store - but can't update container
12:57:25 <jzb> when new etcd comes out, need to pull a new container (and build)
12:57:27 <jzb> simple signing
12:57:32 <jzb> wanting to have concept of simple signing
12:57:37 <jzb> similar to what we use with RPM
12:57:47 <jzb> user can sign document ... "dan walsh trusts this image"
12:58:06 <jzb> "we dont' trust all images by dan walsh, but we'll sign ones we trust"
12:58:11 <jzb> so you add signatures
12:58:28 <jzb> adding concept to these tools, add gpg type signature
12:58:40 <jzb> allow users to pgp sign a lookaside cache
12:58:46 <jzb> allow signatures to be stored anywhere
12:59:03 <jzb> put 'em on a Web site, "there are 40K docker images, but I only trust these 4 - here are their checksums"
12:59:22 <jzb> really simple stuff, not a hierarchy
12:59:29 <jzb> letting people work with signing based on this concept
12:59:33 <jzb> policy rules engines
12:59:37 <jzb> to say who you trust
12:59:50 <jzb> set up a policy, only allow docker engine run applications signed by acme inc
13:00:49 <jzb> introducing the OCID effort
13:01:15 <jzb> look at new tool building called OCID - it's components needed by kubernetes in production
13:01:25 <jzb> container image transport - way to get image to local system
13:01:29 <jzb> need way to store images
13:01:37 <jzb> open container initiative runtime
13:01:41 <jzb> a way to start a container
13:01:50 <jzb> and then I need an api - a wrapper around this application
13:01:58 <jzb> Skopeo - greek for remote viewing
13:02:29 <jzb> used by atomic CLI to view container image data on registry (JSON data)
13:02:43 <jzb> don't have to pull image to look @ json metadata
13:02:59 <jzb> also figured we could add pull/push registry features.
13:03:14 <jzb> worked on splitting library from CLI w/CoreOS
13:03:27 <jzb> skopeo - two parts, cli and library
13:03:34 <jzb> might eventually be used by RKT
13:04:09 <jzb> storage - after pull image, need to be able to store on disk, anybody play with Atomic cli
13:04:16 <jzb> want to look @ content but not run
13:04:34 <jzb> several releases ago, added ability to mount (e.g., "atomic mount fedora /mnt"
13:04:53 <jzb> running containers, can do --live + go into container mount point and then go into container and see content show up
13:06:21 <jzb> graphc, graphtool, cowman, storetool
13:06:40 <jzb> create container storage ClI & library
13:06:48 <jzb> https://github.com/containers/storage
13:07:07 <jzb> tbd - add support ostree, and network storage (Ceph, Gluster, NFS)
13:07:37 <jzb> CoreOS could use it as its backend
13:08:23 <jzb> open container initiative , runc developed in OCI,
13:08:39 <jzb> runc is just an initial implementation of implementation
13:08:51 <jzb> clear linux is about to release a specification to run machines
13:08:58 <jzb> they follow oci spec
13:09:12 <jzb> docker 1.11 uses runc as default backend
13:09:29 <jzb> oci tool https://github.com/opencontainers/ocitools
13:11:26 <jzb> OCID container management API
13:11:36 <jzb> Open Container Initiative Daemon
13:11:40 <jzb> least developed part of project
13:11:50 <jzb> impelments kublet container runtime interface
13:12:02 <jzb> kubernetes / google server interface launching / running pods
13:12:08 <jzb> kubernetes wants to execute
13:12:15 <jzb> kubernetes uses skopeo to pull image
13:12:27 <jzb> ocid stores image on disk using storage
13:12:32 <jzb> then ocid runs container using runc
13:12:47 <jzb> standards based alternative to docker and rkt runtimes
13:12:57 <jzb> want to get into OCI
13:13:04 <jzb> not trying to replace docker
13:13:16 <jzb> trying to supplement -
13:13:36 <jzb> base on kubernetes but implementing same functionality
13:13:39 <jzb> for atomic
13:13:52 <jzb> move openshift to OCID by default
13:14:00 <jzb> and then have that standardized
13:14:18 <jzb> where docker is constantly changing, we dont' have to swap things out based on changes in Docker upstream
13:14:35 <jzb> containers in production will focus on system containers for ... in production
13:15:26 <jzb> non-production container == under development
13:16:56 <nardasev> question: Copr recently sucks
13:17:09 <nardasev> A: we are aware of that and are trying to fix it as hard as we can
13:17:44 <nardasev> huge rebuilds cause problems, some services time out...
13:17:56 <nardasev> we're trying to change the queue logic
13:19:55 <jzb> #endmeeting