19:00:01 <nirik> #startmeeting Infrastructure (2011-12-08)
19:00:01 <zodbot> Meeting started Thu Dec  8 19:00:01 2011 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:01 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
19:00:01 <nirik> #meetingname infrastructure
19:00:01 <nirik> #topic Robot Roll Call
19:00:01 <nirik> #chair smooge skvidal Codeblock ricky nirik abadger1999 lmacken dgilmore mdomsch
19:00:01 <zodbot> The meeting name has been set to 'infrastructure'
19:00:01 <zodbot> Current chairs: Codeblock abadger1999 dgilmore lmacken mdomsch nirik ricky skvidal smooge
19:00:10 * skvidal is here
19:01:03 <nirik> morning everyone.
19:01:10 * nirik will wait a few min for folks to wander in
19:01:26 * jnalley is here
19:01:26 <jac1bat0> hello nirik, everybody
19:01:32 * abadger1999 here
19:01:33 * jsmith is lurking
19:02:08 * mdomsch is lurking too
19:03:08 <nirik> #topic New folks introductions and Apprentice tasks
19:03:13 <nirik> ok, lets dive in.
19:03:25 <nirik> any new folks or apprentices care to discuss anything?
19:03:33 <nirik> or introduce themselves?
19:03:59 <nirik> I'm going to try and add some more easyfix tickets soon... if I can get fires quenched.
19:04:27 <jnalley> hi, i introduced myself on the list a while back but haven't really started working on a ticket since then.  just today i contacted some ticket owners about helping out on otherwise stale tickets that hopefully will be easy for an fi-n00b.
19:04:43 <nirik> jnalley: cool. Welcome.
19:04:50 <jnalley> and it does't help that i also changed my nick from the less-intelligble nick ke4zvu3
19:04:54 <nirik> do hang out in #fedora-admin and/or #fedora-noc and chime in as you can.
19:05:02 <jnalley> i have, do, and will
19:05:05 <nirik> lost your ham license? ;)
19:05:18 <jnalley> haha, no, just changed it to be uniform with my FAS name
19:05:28 <nirik> oh good.
19:06:01 <nirik> any other introductions or issues with apprentice tasks?
19:06:17 * nirik moves on.
19:06:25 <nirik> #topic serverbeach / collab / hosted status
19:06:40 <nirik> so, we have our new new hardware in, we have installed on it.
19:06:51 <nirik> I've sent out an announcement about moving collab on next monday.
19:06:59 <nirik> from collab01 to collab03.
19:07:19 <nirik> Hopefully that will be a pretty easy migration.
19:07:42 <nirik> Also, I am going to send out an announcement later today about moving hosted on next wed... from hosted01 to hosted03.
19:08:10 <nirik> If anyone has ideas for stress testing the new machines before we migrate to them, please do speak up.
19:08:47 <nirik> right now, I am thinking of just moving hosted pretty much as is, because I want to get it off rhel5/old machine.
19:09:04 <skvidal> nirik: maybe just rsync back and forth from the systems
19:09:09 <skvidal> to fill up the disks - to test the range
19:09:19 <nirik> then we can schedule a lists.fedorahosted.org move later along with a more targeted move of projects to a new setup with cnames.
19:09:35 <nirik> skvidal: yeah, I have been running rsyncs... they do have the data from the live/prod boxes on them.
19:10:06 <nirik> I need to finish the migration script for hosted and then we can start hitting it for testing again.
19:10:18 <nirik> (via a /etc/hosts change)
19:11:17 <nirik> also, it's worth noting that I setup hosted03/04 and collab03/04 with drbd... It means if the primary dies, we can bring up the data on the secondary pretty easily and not have much loss.
19:12:00 <nirik> so, any questions or concerns on all that? or shall we move on?
19:12:28 <nirik> #topic ibiblio machines status
19:12:40 <nirik> we have a new ibiblio03 coming in next week or so I think.
19:12:55 <nirik> we have also started moving things from ibiblio01->02.
19:13:04 <nirik> (which ran into some anoying ipv6 pain yesterday. ;(
19:13:38 <nirik> we have 3 things still on 01...
19:13:49 <nirik> smtp-mm03 (which can be migrated in the next few days)
19:14:01 <nirik> backup02 (which I want to migrate to ibiblio03 once it's in)
19:14:05 <nirik> and torrent01.
19:14:16 <skvidal> :( to that last one
19:14:16 <nirik> what will it take to migrate torrents to torrent02?
19:14:24 <skvidal> nirik: a torrent client from this decade?
19:14:39 <nirik> yeah, that would be nice.
19:14:44 <skvidal> we're sorta up shit creek when it comes toa torrent seed and tracker that has the same features
19:14:46 <skvidal> I can get one working
19:14:48 <skvidal> but not the logging
19:14:51 <skvidal> and stats tracking
19:14:58 <skvidal> that btseed/ bttrack currently offers
19:15:06 <nirik> fun. Can we just rebuild the thing we have now for now?
19:15:22 <skvidal> I can probably force it to work on el6
19:15:24 <nirik> and look at moving to magnet or something.
19:15:32 <skvidal> magnet links will provide even less info
19:15:44 <skvidal> it lets us just distribute out the link info - which is nice
19:15:52 <skvidal> but won't help us provide stats on if they are being used :(
19:16:05 <nirik> ok. wasn't herlo investigating some of the options?
19:16:25 <skvidal> nirik: istr, yes - but last I heard the options were all full of suck..
19:16:34 <skvidal> I'll ask him if he had a final report or not
19:16:41 <nirik> worth asking on the devel/infra lists for ideas?
19:16:59 <skvidal> <shrug> I suspect we
19:17:10 <skvidal> 'll get a dozen comments of an unhelpful nature
19:17:18 <skvidal> that's not cynicism - just reality
19:17:20 <nirik> well, I'd like to move things off ibiblio01 so we can rhel6ize it, but we can see.
19:17:28 <nirik> yeah, probibly the case.
19:17:35 <skvidal> it seems like outside of the tpb no one really USES torrents much
19:17:55 <nirik> perhaps we could contact them and get them to open source their stuff? ;)
19:18:00 <skvidal> hahahaha
19:18:03 <skvidal> you're funny
19:18:28 <skvidal> oh
19:18:28 <skvidal> right
19:18:33 <skvidal> I knew I had forgotten something
19:18:43 <skvidal> the other issue we were hoping to work around
19:18:45 <skvidal> torrent + ipv6
19:18:55 <skvidal> which the existing client does not support, at all.
19:19:09 <nirik> yeah. ;(
19:19:25 <nirik> #info gather more info/options for torrent seed/trackers.
19:19:34 <skvidal> if we could dump the data gathering requirement
19:19:40 <skvidal> we could make something else work, probably  quickly, too
19:19:56 <nirik> So what do we use that data for?
19:20:01 <nirik> I guess spins popularity?
19:20:56 <skvidal> and in general
19:21:02 <skvidal> just 'how much stuff got downloaded'
19:21:08 <nirik> well, the statistics page doesn't really use them anymore...
19:21:13 <skvidal> nod
19:21:23 <mdomsch> skvidal: opentracker still not up to snuff?
19:21:25 <mdomsch> I haven't looked in a while
19:21:27 <nirik> but the spins.fp.o uses them a lot
19:21:39 <mdomsch> last I knew, it still had the split v4 and v6 daemons
19:21:41 <skvidal> mdomsch: stats crash iirc
19:21:48 <mdomsch> skvidal: I think that's fixed now
19:21:52 <mdomsch> per bugzilla
19:22:08 <mdomsch> worth a look at least
19:22:23 <skvidal> mdomsch: I think this is what herlo was looking into
19:23:04 <nirik> I guess if it comes down to it, we can just move torrent01 over to ibiblio02 as a rhel5 if we can't come up with any better options.
19:23:44 <nirik> ok, gather more info, try and find a non sucky option...
19:23:46 <skvidal> a question I'd love an answer to
19:23:53 <skvidal> do we really NEED the torrent anymore?
19:24:04 <skvidal> can we just deprecate it? would the sky fall down on us?
19:24:08 <nirik> well, spins are currently not widely mirrored
19:24:13 <nirik> they are torrent only.
19:24:22 <skvidal> torrent _only_?
19:24:23 <skvidal> no
19:24:24 <nirik> (well, they are in the alt mirror component, but very few mirrors mirror that)
19:24:26 <skvidal> I don't think that's true
19:24:28 <skvidal> okay
19:24:31 <skvidal> so not torrent _only_
19:25:12 <nirik> yeah, but a pretty small number of mirrors.
19:25:41 * nirik would be happy to punt that decision to the Board or fesco. ;)
19:25:43 <skvidal> how large is the consume population?
19:26:26 <nirik> from looking at the torrent stats, not all that vast.
19:26:40 <mdomsch> opentracker is in the trees, and is lightly maintained (one spec patch since Jan)
19:27:27 <nirik> lets try and gather more info, and revisit next week...
19:28:03 <dgilmore> hola
19:28:11 <nirik> hey dgilmore
19:28:19 <nirik> #topic Upcoming Tasks/Items
19:28:27 <nirik> flood ahead:
19:28:31 <nirik> #info 2011-12-08 - Fedora 14 end of life.
19:28:31 <nirik> #info 2011-12-12 - migrate to collab03/04 (evening) (tenative)
19:28:31 <nirik> #info 2011-12-14 - migrate hosted (tenative)
19:28:31 <nirik> #info 2011-12-22 - contact peer1 about retirement schedule for old machines.
19:28:31 <nirik> #info 2011-12-23 to 2012-01-02 - rh shutdown week.
19:28:31 <nirik> #info 2012-01-10 - drop inactive maintainers from pkgdb acls.
19:28:33 <nirik> #info 2012-01-13 to 2012-01-15 - FUDCON blacksberg
19:28:37 <nirik> #info 2012-02-31 - fas 0.8.11 final release.
19:28:39 <nirik> #info 2012-02-14 to 2012-02-28 - F17 Alpha Freeze
19:28:48 <nirik> anything anyone would like to discuss upcoming?
19:29:43 <nirik> ok, moving along then.
19:29:47 <nirik> #topic Meeting tagged tickets
19:30:10 <nirik> none.
19:30:21 <nirik> #topic NFS outage
19:30:25 <dgilmore> :(
19:30:28 <nirik> We are currently in a nfs outage. ;(
19:30:41 <nirik> it looks like it was caused by a network hiccup to the backend storage.
19:31:05 <nirik> Not sure what we can do to avoid this kind of thing moving forward, but it's an anoying SPOF
19:31:26 <dgilmore> yeah
19:31:38 <dgilmore> iscsi ftw
19:32:22 <nirik> so, if we have ideas for clustered filesystems or the like, we can investigate. I'm not sure I trust any of them with our data yet tho.
19:32:33 <dgilmore> yeah
19:32:38 <skvidal> all of the clustered fs will be an issue
19:32:43 <skvidal> b/c of ownerships afaict
19:32:47 <nirik> yeah
19:32:49 <skvidal> maybe gfs2
19:32:55 <dgilmore> skvidal: yeah
19:33:03 <skvidal> but I'm not sure that's going to not require an arseload of infrastructure that we do not have
19:33:09 * nirik hasn't looked at gfs2 lately, but long ago I remember it being a royal pain to setup.
19:33:27 <dgilmore> maybe a clusered fs would help with a network glitch
19:33:38 <mdomsch> gluster ? :-)
19:33:44 <dgilmore> but its a whole different ball of fish
19:33:59 * nirik notes that RH announced a storage product today...
19:34:10 <skvidal> gluster
19:34:11 <mdomsch> nirik: yup
19:34:14 <skvidal> gluster has the perms issues
19:34:32 <skvidal> it can do normal posix
19:34:32 <nirik> no acls, right?
19:34:35 <skvidal> but not extended acls
19:34:45 <skvidal> and no selinux - though that may have changed very recently
19:34:50 <mdomsch> ask ric if that's coming...
19:34:52 <dgilmore> we dont do acls
19:34:54 <nirik> not sure we use acls on /mnt/koji
19:35:00 <dgilmore> nirik: we dont
19:35:14 <skvidal> mdomsch: I've asked the gluster folks and it is tricky
19:36:14 <nirik> so, that might be something to look into... would need to setup a testbed and make sure it doesn't blow up and that we understand it.
19:36:51 <nirik> not sure what we could do short term. ;(
19:37:12 <dgilmore> yeah
19:37:20 <dgilmore> not sure it would really help
19:37:32 <dgilmore> gluster is userland right
19:37:40 <dgilmore> on top of something else?
19:37:52 <skvidal> dgilmore: it's userland - yes - but it replicates across the network
19:38:17 <skvidal> dgilmore: you then mount it from userland onto your clients - I want to say it is fuse
19:38:20 <skvidal> on the client end
19:38:22 <nirik> yes, fuse
19:38:27 <dgilmore> skvidal: right
19:38:36 <dgilmore> skvidal: which would do nothing in this case
19:38:44 <skvidal> ??
19:38:46 <skvidal> it would do nothing?
19:38:53 <dgilmore> since the issue was that the iscsi storage blipped
19:38:57 <skvidal> if we had 2 units setup as mirrors
19:38:59 <dgilmore> and ext4 got messed up
19:39:03 <skvidal> and each of them had independent backends?
19:39:17 <dgilmore> skvidal: so if we got another blip likely get the same
19:40:03 <skvidal> oh - if they are both backended on the same storage network
19:40:06 <skvidal> yah - I see
19:40:18 <dgilmore> skvidal: yeah.
19:40:29 <skvidal> so do we have a place to replicate the nfs data to?
19:40:41 <dgilmore> skvidal: bnfs01
19:40:49 <dgilmore> but it takes me a day to rsync to it
19:40:56 <nirik> drbd!
19:40:57 <skvidal> would drbd
19:40:58 * nirik runs
19:41:00 <skvidal> exactly
19:41:03 <skvidal> will it scale to that size?
19:41:09 <dgilmore> 12T
19:41:20 <dgilmore> no idea
19:41:27 <nirik> it can, as long as the net can keep up to the disk i/o
19:41:49 <nirik> which usually isn't a problem
19:42:00 <skvidal> are bnfs01 and koji's nfs backended to different devices?
19:42:08 <skvidal> or is it all onto the netapp?
19:42:08 <dgilmore> yes
19:42:19 <skvidal> yes to which one
19:42:23 <nirik> different devices
19:42:27 <dgilmore> bnfs01 is 1tb drives attached directly using disk shelves
19:42:37 <dgilmore> nfs01 is equalogic over iscsi
19:42:50 <skvidal> and how close to end of life is bnfs01, it's disks, etc?
19:43:03 <dgilmore> both are ~1 year old
19:43:12 <nirik> 2013-01-17
19:43:16 <skvidal> okay, great
19:43:20 <skvidal> wait
19:43:24 <skvidal> Jan 2013?
19:43:29 <skvidal> so a bit over a year?
19:43:33 <skvidal> like 13 months?
19:43:34 <nirik> yep
19:43:39 <skvidal> bleah
19:43:42 <dgilmore> nirik: that doesnt seem right
19:43:43 <nirik> but we can extend.
19:43:54 <nirik> unless it's labeled wrong. it could be.
19:44:07 <jds2001> arent they normally 3 years?
19:44:08 * nirik can check for sure.
19:44:14 <nirik> jds2001: yep
19:44:18 <dgilmore> i thought normally 5 years
19:44:21 * jds2001 checks into the cheap seats :)
19:44:24 <skvidal> jds2001: well if ~1yr old == 2yrs
19:44:24 <nirik> 3 is normal
19:44:30 <skvidal> jds2001: then.. :)
19:44:43 <jds2001> heh
19:44:49 <nirik> in any case... one pain of moving to drbd is that we need somewhere to shuffle the data, make the drbd and then shuffle back. ;)
19:44:49 <dgilmore> im often wrong
19:44:57 <skvidal> nirik: nod
19:45:13 <skvidal> nirik: we could probably beg for tmpspace on the netapp
19:45:16 <skvidal> nirik: for a few weeks
19:45:17 <nirik> yeah.
19:45:39 * nirik has used heartbeat+drbd for nfs servers in the past.
19:45:52 <skvidal> you'd have to remount everything
19:46:00 <skvidal> b/c the filehandles would all change
19:46:05 <nirik> yeah.
19:46:07 <skvidal> it's doable - but with our static mounts its harder
19:46:09 * skvidal likes autofs
19:46:20 <skvidal> but I'm probably one of the 4 people in the universe who does
19:46:29 <jds2001> skvidal: that makes one of us :)
19:46:43 <nirik> anyhow, thats also longer term... perhaps we could have a nice 'nuke SPOFs' talk at fudcon too.
19:46:54 <skvidal> nirik: we had that talk at the last fudcon :)
19:47:11 <nirik> yeah, and we can keep doing it until we get it right! :)
19:47:19 <skvidal> hah
19:47:22 <skvidal> oh and remember everyone
19:47:28 <skvidal> bring your body armor to fudcon
19:47:34 <skvidal> http://www.cbsnews.com/8301-201_162-57339465/2-shot-dead-at-va-tech-gunman-sought/
19:47:40 * jds2001 prepares for the 'nuke SPOF' talk at fudcon 2060 :D
19:47:41 <nirik> yeah, crazy.
19:47:49 <nirik> anyhow...
19:47:49 <dgilmore> skvidal: im all for eliminating spof
19:47:56 <nirik> #topic Open Floor
19:48:05 <nirik> anyone have anything for open floor?
19:48:48 * jds2001 hasn't been around in infra a ton lately
19:48:53 * jds2001 apologizes :/
19:48:55 <nirik> jds2001: welcome back. ;)
19:49:05 <skvidal> there's a lot of tasks
19:49:07 <skvidal> which need doing
19:49:11 <skvidal> which need time
19:49:16 * nirik gets the list of things we assigned to jds2001 while he wasn't around. ;)
19:49:18 <skvidal> if anyone wants to work on projects
19:49:21 <skvidal> there are a bunch of them
19:49:50 <jds2001> well all of this talk of collab and mailing lists is near and dear to my heart.
19:50:20 <jds2001> though i really dont want to work on it for 14-18 hours straight like last time :)
19:50:28 <nirik> we still do need to get the last lists at rh moved too someday.
19:50:37 <skvidal> nirik: like what?
19:50:45 <jds2001> epel and such
19:50:46 <nirik> epel-devel, some of the ambassadors lists.
19:51:26 <nirik> I am hoping the collab move will be very easy. I have fewer hopes for the fedorahosted one...
19:52:00 <jds2001> well, the hosted move is similar to moving from rh, right?
19:52:10 <abadger1999> nirik: Do you want to update on what bressers said?
19:52:13 <jds2001> that wasn't terribly hard, except a few glitches.
19:52:44 <nirik> jds2001: the moving lists.fedorahosted -> collab? yeah, that will be just like that, but with another domain tossed into the mix.
19:52:46 <abadger1999> nirik: I was wondering specifically if he'd said anything about how much diversity wold be good/bad.
19:52:54 <nirik> abadger1999: oh yeah, sorry, I meant to mention that.
19:53:14 <jds2001> nirik: and having control of both sides makes it even easier, really.
19:53:16 * abadger1999 is sure that 2 chars is good, but unsure above that.
19:53:40 <nirik> so, he said:
19:53:45 <nirik> "I would say in the short term, enact some of the above rules. They won't
19:53:46 <nirik> hurt and will certainly prevent really bad passwords. They won't solve the
19:53:46 <nirik> real problem though."
19:54:04 <nirik> sadly, he is not going to be at fudcon. ;(
19:54:17 <nirik> perhaps we could invite him for next weeks meeting and discuss this a bit more?
19:54:41 <abadger1999> Sure.
19:55:13 * nirik sees if he's around now. guess not.
19:55:22 <skvidal> abadger1999: what is 'the real problem'?
19:55:41 <nirik> skvidal: having passwords instead of 2factor or the like.
19:55:45 <skvidal> ah
19:55:57 <skvidal> I do have an update on the 2fa stuff
19:56:22 <skvidal> I worked on the pam_linotp module - I got it building and it submits the 2fa input over the wire to a cgi which can verify it
19:56:46 <nirik> cool.
19:56:53 <skvidal> so that works just fine. to make 2fa happen on EVERY ssh login, though, we'll need to modify sshd
19:57:00 <skvidal> with patches from dwmw2
19:57:18 <skvidal> if we just want them on sudo, though, that should be fine right now
19:57:28 <nirik> so what can be used as the other factor there? googleauth? yubikey?
19:57:51 <jds2001> yubikey is simple and i already have one :)
19:57:53 <skvidal> right now it would be totp
19:58:14 <skvidal> but, ostensibly, you could verify whatever in the cgi
19:58:17 * jds2001 hasnt even looked at googleauth, but it seems cool.
19:58:35 <skvidal> since the string sizes are so massively different between totp and yubikeys
19:58:35 <skvidal> it should be possible to accept either
19:58:48 * dgilmore doesnt trust anything with google in the name
19:58:57 <nirik> dgilmore: it's open source. ;)
19:58:59 * skvidal shakes his head
19:59:13 <nirik> I'd be find with yubikeys for sudo, but want a pin + yubikey
19:59:20 <jds2001> dgilmore: at least its not cubsauth :D
19:59:20 <dgilmore> nirik: i know, I have an irrational distrust of anything google does
19:59:23 <dgilmore> open or not
19:59:25 <nirik> and it sounds like this linotp might be able to let us do whichever
19:59:33 <skvidal> nirik: all linotp offers is this
19:59:43 <skvidal> 1. a pam_module to submit the info to a cgi elsewhere
19:59:48 <abadger1999> I think having ssh key for login and 2fa auth for sudo would offer a lot over what we currently have.
19:59:52 <dgilmore> jds2001: id be ok with that
20:00:11 <nirik> abadger1999: +1. I'd like that
20:00:12 <skvidal> 2. some simple routines to verify various types of 2fa
20:00:28 <skvidal> to make something work for both totp and yubikey
20:00:35 <skvidal> we'd have to maintain it ourselves
20:00:43 <skvidal> though I suspect
20:00:45 <skvidal> if we do this
20:00:51 <skvidal> other people would be interested
20:00:57 <nirik> yeah.
20:01:21 <dgilmore> skvidal: i agree
20:01:41 <skvidal> modifying the pam module to get it  working took some minor patching - nothing serious
20:01:49 <skvidal> and there are a few things in the module which make me grumpy
20:01:51 <nirik> is it packaged up at all?
20:01:57 <skvidal> oh god no
20:02:04 <nirik> ok.
20:02:06 <skvidal> and to be fair
20:02:15 <skvidal> packaging up all of lin_otp seems like a bad idea
20:02:23 <skvidal> they want to be a one-stop shop for all sorts of auth stuff
20:02:25 <nirik> ok, we just need the pam module?
20:02:29 <skvidal> and integrating it into fas would be a mess
20:02:47 <skvidal> pammodule + whatever cgi the pam module submits info to
20:03:06 <skvidal> they have a cgi which is django based and involves all their other admin-interface
20:03:07 <nirik> yeah.
20:03:15 <skvidal> which would probably make us want to kill ourselves to keep it going
20:03:25 <nirik> right, so perhaps we want to look at forking and cleaning up/simplifying this.
20:03:30 <skvidal> right
20:03:55 <skvidal> so let me confirm a few things
20:03:57 <nirik> abadger1999: you had some questions for bress ?
20:03:57 <skvidal> b/c that will help
20:04:02 <jds2001> has anyone had disucssions with upstream about that?
20:04:14 <jds2001> i.e. making a "lin_otp lite" if you will?
20:04:18 <skvidal> jds2001: they are using the split-source/commerical modules thing
20:04:25 <jds2001> :/
20:04:35 <skvidal> jds2001: so... no
20:04:56 <skvidal> oh and one more hitch to using a lot of their code
20:05:03 <skvidal> some of it is agpl
20:05:05 <abadger1999> bress: hey, so about checking password strength by eliminating passwords of only a few different characters.... is there some number of different characters that makes things worse instead of better?
20:05:06 <nirik> ugh
20:05:32 * nirik notes we are over time, but I don't think there is a meeting after us if we want to keep going.
20:05:46 <skvidal> I'd like to get confirmation on a few things
20:05:53 <skvidal> nirik: after abadger1999 and bress are finishe
20:05:54 <skvidal> d
20:05:57 <nirik> skvidal: ok
20:07:03 <abadger1999> bress: My back of the envelope calculation is ensuring two separate characters for standard passwords has little drawback -- but my math skills aren't up to tackling how larger diversity requirements affect the ability to brute force passwords
20:07:03 <bress> abadger1999: I mailed Kevin Fenzi about this a while back. In general, password policies make things worse. The real answer is some sort of 2 factor system. In general though, you're not talking about reducing your corpus by all that much.
20:07:16 <abadger1999> k.
20:07:20 * nirik is kevin, FYI. ;)
20:07:24 * jds2001 thinks that no matter what technical restrictions we put on passwords, we'll always have someone choose something like AbC123$
20:07:31 <abadger1999> <nod>
20:07:39 <skvidal> jds2001: shit, now I have to go change my password!
20:07:46 <nirik> the more restrictions also the worse documentation looks
20:07:50 <bress> I wouldn't be as worried about someone using a shitty password as I would be making sure an attacker can't brute force any of our systems. If you allow more than say 6 logins per minute, we're doing it wrong.
20:08:09 <bress> skvidal: Just go with AbC1234$
20:08:14 <nirik> your password must be at least 12 characters, unless it's monday in which case you need to use 2 non repeating lower case letters, unless it's a full moon, in which case...
20:08:22 <StylusEater> bress: +1 on the login comment ... ;-)
20:08:30 <abadger1999> bress: So ensuring diversity of characters is going to hit, "what will our users stand for" quicker than "you've reduced your possible combinations too mch".
20:08:40 <bress> abadger1999: Yeah
20:08:48 <abadger1999> bress: thanks.
20:08:50 <bress> abadger1999: It could be wise to point folks at diceware.
20:09:07 <bress> Since picking a new password is really hard for people.
20:09:31 <nirik> yeah, we did link to that in our change announcement.
20:10:03 <skvidal> abadger1999: how hard would it be to modify our fas web logins so we require password + 2fa?
20:10:07 <bress> But I'd say the real answer here is going to be some sort of 2 factor auth.
20:10:29 <abadger1999> bress: That's nice -- smooge had also suggested ading javascript to our password reset that does something similar.
20:10:42 <abadger1999> As a password suggestion for people.
20:10:45 <smooge> working on that
20:10:46 <abadger1999> <nod>
20:10:48 <bress> Yeah, that's a good idea.
20:11:04 <smooge> sorry for missing the meeting guys
20:11:13 <skvidal> bress: which places do you think 2fa should be used: on any ssh login? git commits? webapp logins? sudo?
20:11:26 <nirik> smooge: lucky for you, it's still going on. ;)
20:11:42 <abadger1999> skvidal: "allow" pretty easy -- it's a variant on the yubikey acceptance that's already in.
20:11:54 <abadger1999> skvidal: "require" is harder.
20:11:56 <bress> skvidal: That one is tricky. I'd probably say anywhere a password is needed. Things where an SSH key will do is probably OK.
20:12:05 <abadger1999> skvidal: "require for some people" even more.
20:12:08 <bress> skvidal: If people can't commit without jumping through hoops, they just won't commit.
20:12:39 <skvidal> bress: oh cmon - marriage rates alone disprove that <zing>
20:12:46 <bress> The trick isn't to ellimate risk, but to understand and control it.
20:13:05 <skvidal> so another question wrt to commits
20:13:10 * bress wonders how long skvidal has been waiting to use that one ;)
20:13:12 <smooge> so question. To get an idea of how bad our users practices are.. can we "anonymize" the hashes from before the new hash became standard and run jtr with rockyou top 10000
20:13:15 <abadger1999> skvidal: but none of it is something we lack the ability to do (even considering how time constrained we are)
20:13:43 <skvidal> would it help us to require people to connect with an ssh master connection - requiring 2fa for that connection duration
20:13:53 <smooge> then if we find a large number before and after we say "simple test" if you chose a password here you need to chose something else.
20:13:59 <skvidal> so they could connect once, stay on, and then not have to auth again until forced out
20:14:12 * skvidal is just brainstorming a bit
20:14:37 <nirik> I suspect we will get a lot of pushback on making pkg commits 2factor.
20:14:46 <bress> skvidal: That's an interesting concept. The only problem I can see is people forgetting to logout (or just never logging out to save time)
20:14:46 <skvidal> nod
20:14:55 <skvidal> bress: forgetting to logout we can actually handle
20:15:09 <skvidal> bress: in much the same way rh handles it when I neglect to disconnect from the vpn
20:15:10 <bress> nirik: Ahh, but we can make them 2 factor sneakily. Do ssh for auth, and have users sign their commits :)
20:15:18 <jds2001> the issue that i see is like bress said, we're making things harder.
20:15:20 <bress> skvidal: You disconnect from the VPN? :)
20:15:27 <skvidal> bress: it makes me
20:15:35 <skvidal> bress: every 10hours or so
20:15:39 <bress> skvidal: I can keep openvpn up for days.
20:15:44 <bress> skvidal: But that's another conversation.
20:15:47 <skvidal> yes
20:15:48 <skvidal> anyway
20:15:50 <skvidal> my point is
20:16:00 <skvidal> we could force all communication through a host (or a set of hosts) like fedorapeople
20:16:01 <nirik> yeah, signing commits would be nice/cool.
20:16:18 <skvidal> and make everyone go through a 2fa there first
20:16:53 <skvidal> we could kill connections > N hours/days old
20:17:24 <skvidal> nirik: as far as signing commits - are we expecting gpg signs? and what about systems not using git? ie: fedorahosted?
20:17:42 <abadger1999> git and bzr both support gpg sig
20:17:53 * abadger1999 doesn't know about hg ... svn doesn't
20:18:00 <nirik> I was mostly waiting on the fallout from the kernel.org git commit signing stuff... which as far as I know hasn't really reached a conclusion.
20:18:04 <bress> We'd be crazy not to sign commits. It gives so many levels of safely it's not even funny.
20:18:06 <nirik> (but it might have and I missed it)
20:18:57 <skvidal> well we're going to need gpg keys from everyone and we have A LOT more people and a lot less tightly knit people than the kernel devs
20:19:06 <nirik> I think we can get short term win by inflicting 2f on smaller groups (sysadmin-main, then possibly sysadmin*), then learn from that before trying to deal with any larger groups.
20:19:19 <skvidal> okay
20:19:38 <nirik> if we end up using gpg more, we could require it for some groups?
20:20:03 * jds2001 was required to use GPG to sign the CLA back in the day
20:20:10 <skvidal> so I grok how gpg helps for commits
20:20:17 <skvidal> but it does nothing for system security afaict
20:20:21 <jds2001> one of hte main reasons for changing that was the technical chanllenges of doing that.
20:20:24 <bress> We don't require gpg keys from everyone?
20:20:30 * bress had no idea
20:20:39 <nirik> bress: fas has a field for it, but there's no requirement for it as far as I know.
20:20:45 <abadger1999> jds2001: Well... you had to sign the cla to edit the wiki so....
20:21:05 <abadger1999> yeah -- also, they're gpg fingerprints, not keys
20:21:13 <nirik> yeah, right.
20:22:08 <bress> Anyway, this conversation has gotten derailed. I guess to sum it up, a simple password policy is best, and should help get us to a point where two factor auth can be used.
20:22:42 <nirik> my thoughts: short term try and get linotp or something flexable based of it going... then try and modify fas to handle yubikey+pin, and/or googleauth and/or whatever, then use that for sudo at least, and move from there.
20:23:03 <bress> abadger1999: Also, there's nothing wrong with adding rules to the code and not really telling people. I don't think anyone is going to complain by saying "When I tried all A's it didn't work", they're look like an idiot :)
20:23:27 <abadger1999> bress: Now that's something I kinda like ;-)
20:23:55 * nirik notes if we had yubikey + pin, he would be ok with using that for sudo now, but we don't.
20:24:14 <smooge> as long as it allows hellokittyhellodoggiewhysoserious
20:24:38 <bress> smooge: You probably need to add a 4 to the end to make it secure.
20:24:59 <skvidal> nirik: okay
20:25:08 <skvidal> so the confirmations I needed are mostly answered
20:25:14 <skvidal> but I need a couple more details
20:25:17 <abadger1999> bress: Somewhat side note -- is it okay for us to send you best practice security questions?  We do get requests to implement things that I have no idea how to evaluate the impact from time to time.
20:25:19 <nirik> fire away.
20:25:33 <bress> abadger1999: Certainly. I'm happy to help any way I can.
20:25:41 <skvidal> abadger1999: do we have a place we could stuff OTP "secrets"
20:25:50 <skvidal> abadger1999: in fas, or something related, I mean.
20:26:05 <abadger1999> bress: Excellent.  I'll make use of your expertise I'm sure :-)
20:26:24 * nirik is sad bress isn't going to make it to fudcon. ;(
20:26:39 <bress> abadger1999: Heh, sure, that's what I call it too :)  But if I don't know, I probably know someone who does.
20:26:42 <abadger1999> skvidal: We can push the shared secret into fas's config table like the yubikey stuff has.
20:26:46 <skvidal> bress: wrt to totp 2fa - do you know if it is even theoretically possible, given enough valid entries to take time + valid OTP and get the secret out of it?
20:28:15 <bress> skvidal: I can't answer that really. I've not seen much research done in that area. I would presume it's *possible* but probably not very practical.
20:28:20 <nirik> there was also a lwn article this week on google authenticator. There was mention there of a command line tool too. (although it required converstion)
20:28:23 <skvidal> bress: second question - if we wanted to get a pam module audited for sanity, etc - can we send it to you
20:28:30 <skvidal> nirik: the command line tool is trivial
20:28:32 <skvidal> nirik: so here's the thing
20:28:42 <skvidal> totp is really easy to implement
20:28:54 <skvidal> the only tricky bit that google has implemented is their secret bailout codes
20:28:57 <bress> skvidal: Sure, it would give me a good opportunity to work out some of the bugs in my code auditing guide.
20:29:19 <skvidal> and afaict the only thing they've done is just keep the list of bailout codes and check against them - there's no math - they just look them up
20:29:21 <bress> skvidal: Google has made a google authenticator pam module public.
20:29:29 <skvidal> bress: yes - and we can't use it
20:29:35 <skvidal> bress: it only auth's locally
20:29:39 <nirik> well, we don't want to use it. ;)
20:29:42 <bress> Yes.
20:29:47 <skvidal> which means we'd have to haqve secrets on EVERY machine
20:29:56 <skvidal> so what I found was this module called pam_linotp
20:30:08 <skvidal> and all it does is relay your otp onto a central cgi
20:30:18 <skvidal> which checks it and gives you a yes or no or 'error'
20:30:23 <skvidal> and that's it.
20:30:32 <bress> Ooohhh
20:30:32 <skvidal> it relays it using libcurl + ssl
20:30:44 <bress> That part sounds ugly
20:30:47 <skvidal> no kidding
20:30:48 <nirik> much better for our use case at least.
20:31:00 <skvidal> bress: but it';s not really worse than ldap auth when you think about it.
20:31:01 <nirik> (then we only need secrets on a few high security machines)
20:31:08 * skvidal points at nirik
20:31:09 <skvidal> exactly
20:31:18 <skvidal> then we have to bottle up the boxes with access to the secrets
20:31:34 <bress> skvidal: ldap doesn't run curl :)
20:31:42 <skvidal> bress: well this doesn't RUN it either
20:31:44 <skvidal> it uses libcurl
20:31:53 <bress> OK, that's a lot better.
20:31:55 <skvidal> but this is why  Iwas asking about auditing the pam module
20:32:12 <skvidal> it has options to the module for ssl verify, etc
20:32:13 <bress> Yeah, giving it a good beating is probably in order.
20:32:33 * jds2001 has to run to a $DAYJOB meeting
20:32:36 <skvidal> I've done some curl programming before and setting up things like client side ssl certs for connecting to the cgi is pretty straightforward
20:33:37 <nirik> so we need a backend part that has a web interface right (or is in fas, but perhaps seperate would be better). so you could enroll new 2fa thing, setup pin, lost factor, etc... right?
20:33:44 <skvidal> nah
20:33:51 <skvidal> nirik: you'd do the web interface bit in fas
20:34:01 <skvidal> and this would just be able to talk to the fas db which has all the info
20:34:10 <skvidal> or hell, it could even just get a replicated copy every N minutes
20:34:14 <nirik> well, the advantage of non fas is that we could possible get other projects interested in helping/using it...
20:34:21 <skvidal> agreed
20:34:26 <skvidal> hence - replicated copy of the data it needs
20:34:29 <nirik> or... stupid idea:
20:34:32 <skvidal> but use fas for input
20:34:38 <skvidal> nirik: or big pile of files :)
20:35:06 <nirik> on the auth server, chain the other auth method? ie, have yubikey, googleauth local to that machine to run the check, and return to the linotp thing?
20:36:00 <skvidal> huh?
20:36:15 <nirik> yeah, nevermind.
20:36:21 <skvidal> you could make the pam_unix check required - for normal 'checking your password' thing - then have it prompt for the otp
20:36:32 <skvidal> just like a 2fa gmail login does now
20:36:42 <nirik> yeah.
20:36:46 <skvidal> which has the unfortunate result of allowing you to know if you have the right password
20:37:02 <nirik> anyhow, anything else we want to hash out now? or shall we call it a long meeting?
20:37:25 <skvidal> bress: what address for you? bress@rh?
20:37:32 <skvidal> nirik: yah - I think that's all I needed
20:37:40 <bress> skvidal: bressers@redhat.com
20:38:08 <nirik> cool. Hopefully we can setup something nice and flexable.
20:38:15 <nirik> Any other open floor items?
20:38:50 <nirik> ok, thanks for coming everyone!
20:38:53 <nirik> #endmeeting