20:07:28 <ianweller> #startmeeting
20:07:28 <zodbot> Meeting started Sat Dec  5 20:07:28 2009 UTC.  The chair is ianweller. Information about MeetBot at http://wiki.debian.org/MeetBot.
20:07:28 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
20:07:41 <ianweller> #meetingtopic Can't we all just get along? - Sysadmin & Developer Panel
20:08:04 <ianweller> #topic Waiting for our presenters
20:08:09 <mchua> ianweller: thanks! I'll chime in as I can, I have some notes from last session to clean up first. I'll ping you when I'm ready
20:08:16 <ianweller> mchua: noted, thanks
20:08:51 <ianweller> <smooge> hello, my name is stephen, we only use first names here, and i'm a sysadmin
20:09:05 <ianweller> i've been a sysadmin since 1984, been recovering last 6 months ago, since i started working for fedora infra and i don't do anything now
20:09:22 <ianweller> the projector is cleaning the air filter
20:09:44 <ricky> Hehe
20:09:49 <ianweller> chairs are being rearranged
20:09:56 <heffer> laughing
20:09:59 <ricky> It's almost like I'm really there!
20:10:10 <ianweller> ricky: :D
20:10:21 <ianweller> ok mdomsch, J5, skvidal, lmacken, mmcgrath are sitting down
20:10:23 <ianweller> toshio is up
20:10:35 <ianweller> smooge is joining the table.
20:10:45 <ianweller> <toshio> this is a panel discussion so mostly we'll be taking questions from you ugys
20:10:49 <mchua> we miss ya, ricky.
20:10:52 <ianweller> in infra we have a lot of hackers and a lot of sysadmins and a lot of both
20:10:57 <ianweller> and it takes a lot for us to get a long
20:10:59 <ianweller> along*
20:11:02 <ianweller> different prioritie
20:11:03 <ianweller> s
20:11:11 <ianweller> so this is a Q&A
20:11:19 <ianweller> (questions are welcome from the internets!)
20:11:25 <ianweller> toshio is introducing himself now
20:11:32 <ianweller> mdomsch is now, he works at dell
20:11:47 <ianweller> <mdomsch> when i'm not doing work i'm getting paid for i'm working late at night on f-infra stuff
20:11:50 <ianweller> i wrote mirrormanager
20:12:10 <ianweller> <J5> i'm john palmieri, i work in infra on fedora community, moksha stuff
20:12:16 <ianweller> now working on developing messaging buses
20:12:26 <ianweller> <skvidal> i'm seth vidal, i do yum and fedora people
20:12:30 <ianweller> (<mmcgrath> sucker)
20:12:48 <ianweller> <mmcgrath> i'm infra team lead, a year ago when the shit hit the fan, it got all over me
20:12:54 <ianweller> i wrote smolt
20:12:57 <ricky> Haha
20:13:10 <ianweller> i had a dream about this event, and it was in the style of a don sween (checking spelling, standby) musical
20:13:16 <ianweller> smooge is singing
20:13:23 <ianweller> ok don sweeney
20:13:46 <ianweller> <smooge> o/` i'm stephen smoogen, i don't do much except heckle the crowd o/`
20:14:21 <ianweller> we are now discussing how old we are
20:14:29 <ianweller> <lmacken> i'm luke macken, i'm the young guy, i'm a developer
20:14:49 <ianweller> i wrote bodhi (the updates system), helped work on the fedora community developer dashboard, play sysadmin every now and then
20:14:55 <ianweller> work on SELinux deployment in our infrastructure
20:15:01 <ianweller> <toshio> any questions to start us off?
20:15:25 <ianweller> <remi> really quickly, one of the topics that we're going to be talking about is just the FOSS process. how do you guys go around creating FOSS people?
20:15:39 <ianweller> <mmcgrath> we're the people to talk to, even though some have created a free environment, we've really embraced it
20:15:51 <ianweller> <mmcgrath> even though our puppet configs aren't completely available, we're working on it
20:15:55 <ianweller> <mmcgrath> we have a FOSS only rule
20:16:15 <ianweller> <mmcgrath> the way we do the process works pretty well, we make a lot of mistakes that are in the open too, we learn from it, i think everybody at these tables here when put under a public scope will rise to the occasion
20:16:21 <ianweller> many would get defensive and angry
20:16:27 <ianweller> we all take criticism seriously, but not too seriously
20:16:38 <ianweller> but we find a good balance, i can't think of anything that has gone in or not gone in because we're all butting heads about i
20:16:41 <ianweller> t
20:16:49 <ianweller> other than something we desperately hate that's there
20:16:57 <ianweller> (smooge coughs "bodhi", lmacken laughs)
20:17:13 <ianweller> <mmcgrath> even though luke is working on rewriting bodhi, my job is to make sure we are still getting bits in and out
20:17:29 <ianweller> whether it's CVSor not, doesn't matter, as long as people are working on things
20:17:43 <ianweller> really, anybody could moveus away from CVS, i think jkeating will be doing it
20:17:57 <ianweller> <toshio> now you and luke both have created software that's been more general, you wanna talk about that?
20:18:15 <ianweller> <matt domsch> rpmfusion has adopted mirrormanager, several projects have approcached me about doing that sa well
20:18:31 <ianweller> there's a lot of fedoraisms in the code, i didn't realize that until someone else used it for their own project
20:18:39 <ianweller> people have been sending me patches, which has been great!
20:18:55 <ianweller> it's nice to see something suited for one project and having it work in many places
20:19:14 <ianweller> <smooge> having been the mifrror manager from long long long ago, it's swonderful having someone outside redhat doing it being ale to be open about it when there's a lot of conflicts at times.
20:19:21 <ianweller> (man i'm good at typos -- ian)
20:19:28 <ianweller> <smooge> here's the thing -- it's worked really really well
20:19:46 <ianweller> <mdomsch> the whole private mirror thing is cool. now you can set it up with mirror manager and it Just Works(tm) when clients put im their fedora CD
20:19:49 <ianweller> in*
20:20:07 <ianweller> <mmcgrath> for those of you who don't know it, mirrormanager is the most feature rich mirror management system BY FAR
20:20:18 <ianweller> <warthog9> mirrormanager is *the* best system out there, nobody else can even touch it
20:20:29 <ianweller> this is a testament to matt, when something goes wrong, i just muck with it
20:20:46 <ianweller> i don't wana bash on centos or anything, but you send mail to the mailing list and when they get around to it, they get around to it
20:20:55 <ianweller> <mmcgrath> in other software luke has moksha being used all over
20:21:10 <ianweller> <lmacken> what we do in FOSS is we solve problems for ourselves, and we get people using it
20:21:25 <ianweller> <lmacken> we're solving problems people need solved, and we're getting communities of people to test, bodhi is an example
20:21:46 <ianweller> <lmacken> i wrote bodhi based on fedora-specific requirements, but we iterate over and over and we make things more generic and usable
20:22:01 <ianweller> #topic what are some of the conflicts that we've had in infrastructure?
20:22:08 <ianweller> <mdomsch> mod_python vs. wsgi
20:22:39 <ianweller> <toshio> we had some things under mod_python and then we generated new apps, and instead of mod_python, we wanted to use mod_wsgi, but they won't exist in the same apache process together
20:22:39 <ianweller> so we had conflicts
20:22:45 <ianweller> <lmacken> couldn't even be installed on the same systems
20:23:18 <ianweller> <mmcgrath> we have app servers that we scale out all the time, so we've already commited to mod python only to find out that it was a mistake -- although mod_wsgi didn't exist at that point
20:23:28 <ianweller> mod_python had really bad memory issues, we were running into memory issues
20:23:35 <ianweller> apps would take each other out
20:23:47 <ianweller> wsgi takes care of all of it, it cheats because it doesn't have long standing processes, but we got better speed out of it
20:23:57 <ianweller> turbogears works pretty seamlessly wit hit
20:24:04 <ianweller> <lmacken> fedora is probably the largest adopter of turbogears
20:24:11 <ianweller> <lmacken> it's paid off, we're using it all over
20:24:20 <ianweller> <mmcgrath> we've had a ton of developers and apps that are TG apps
20:24:26 <ianweller> <mdomsch> the least quality part of MM is TGQ!
20:24:27 <ianweller> TG*
20:24:33 <ianweller> <skvidal> TG is a good example of a problem area
20:24:44 <ianweller> <skvidal> TG is a *horribly* moving target for anything new being developed, has been for 1.5 years
20:24:56 <ianweller> <mmcgrath> this is an issue conflict we brought on ourselves, in fedora the project, we tend to be early adopters
20:25:07 <ianweller> <mmcgrath> we make a lot of mistakes, but we have smart people, we can mitigate them and they go away very quickly
20:25:24 <ianweller> <mmcgrath> i think the core conflict has been the difference between early adoption software and mitigating all the risks that come with that
20:25:28 <ianweller> we have to ask, 'will it disappear/"
20:25:40 <ianweller> <toshio> some things have disappeared from TG even
20:26:01 <ianweller> (talking about something MM relies on that's died in the TG community)
20:26:09 <ianweller> <mmcgrath> it's because a better alternative came up
20:26:25 <ianweller> <mmcgrath> it's more of a framework that grabs best-of-breed stuff and best-of-breed stuff is changing all the time
20:26:33 <ianweller> seth's statement is still valid
20:26:37 <ianweller> most people on TG1 use alchemy, right?
20:26:52 <ianweller> <lmacken> well 1.0.9 is (something not alchemy)
20:27:00 <ianweller> <mmcgrath> but 1.1.0 is sqlalchemy.
20:27:09 <ianweller> it's not like suddenly all of our TG1 apps are TG2
20:27:24 <ianweller> when we look into things like moksha and fcomm, it uses a lot of brand new technologies that nobody's really used
20:27:42 <ianweller> my job as a sysadmin is that i have to learn it, since i have to support/troubleshoott it
20:27:45 <ianweller> but my same concerns exists -- will this still be around, is this a moving target
20:27:58 <ianweller> they still rely on a core technology, SQLobject, which is *deprecated* and might have security issues soon
20:28:13 <ianweller> <toshio> in fact parts of TG1 are not supported upstream anymore
20:28:34 <ianweller> <skvidal> there's too many stacks
20:28:44 <ianweller> i.e., TG2 -> moskha -> fedora community
20:28:53 <ianweller> so when it breaks, you have 3 stacks. which means you have to find the developer.
20:29:21 <ianweller> when things develop, it's trying to make sure that if those things are deployed -- and even if they're not deployed as production, but people are using them -- they need to be able to be troubleshot by people who are dealing with it falling over
20:29:39 <ianweller> that's the heard part because anyone who is following any of the tools will have a hard time trying to figure out where to fix bugs
20:29:54 <ianweller> the rate at which you're developing new and interesting code, you can't use google to solve your problems
20:30:13 <ianweller> chances are, the docs are all you've got!
20:30:29 <ianweller> <lmacken> we're adopting these things really fast but we're heavily involved with the projects we're adopting early on
20:30:35 <ianweller> if TG was a black box to me we wouldn't switch to it
20:30:56 <ianweller> <mdomsch> it's a black box to me but it's fine
20:31:00 <ianweller> since lmacken is there upstream
20:31:07 <heffer> how do you deal with work assignments?
20:31:15 <ianweller> #topic how do you deal with work assignments
20:31:24 <ianweller> <mmcgrath> we've solved it fairly well
20:31:30 <ianweller> in most environments we've been in, we have clear deliniation
20:31:49 <ianweller> in fedora, we have responsibility deliniated pretty well, but if i were to fix bodhi and submit a patch, that's not out of line at all
20:32:07 <ianweller> he was trying to restart MM since it busted for some reason, that's not out of line
20:32:23 <ianweller> we have lines of responsibility for who needs to get work done but people are helping each other out.
20:32:45 <ianweller> as far as the details of that, i'm with mdomsch, something got updated, and toshio/lmacken knows what it is, but i don't
20:32:51 <ianweller> (the role dispatcher)
20:32:56 <ianweller> we have two guys on staff that do, so we're fine
20:33:15 <ianweller> <?> is it a culture or a policy thing
20:33:19 <ianweller> <mmcgrath> it's a policy thing
20:33:29 <ianweller> <mmcgrath> we've got toshio and luke, tey're basically full time developres
20:33:36 <ianweller> matt is a volunteer developer
20:33:38 <heffer> ? = adam williamson
20:33:43 <ianweller> thx
20:34:03 <ianweller> <mmcgrath> if matt were to stop developing for whatever reason, at my level i could probably work with it
20:34:42 <ianweller> <toshio> there are problems, amber was supposed to be our end-user package browser, there was one developer who was working on it and i was talking with him extensively
20:34:52 <ianweller> and we kinda designed it together, but he was the person doing it, and he left fedora
20:34:55 <ianweller> the code died. period, it was gone
20:35:20 <ianweller> this is a couple years later now, and they wanted to get stuff in fcomm that does stuff, and i wanted to get stuff on the srpm level done
20:35:44 <ianweller> and when someone from GSOC came and wanted to help, we said that he was going to work on it in fcomm or something we have now
20:35:58 <ianweller> <mmcgrath> in our environment, we're unique -- part luck, part policy -- we've really embraced python
20:36:19 <ianweller> in addition to that, luke started using TG and we embraced it
20:36:38 <ianweller> we are unique to other environments where there are other developers working on different languages
20:36:50 <ianweller> <skvidal> downside -- when people present a ruby app, there's a lot of resistance
20:36:58 <ianweller> (talk about puppet, and how we dislike java)
20:37:06 <ianweller> (mediawiki is PHP, jds2001 notes)
20:37:32 <ianweller> <mmcgrath> i fervently resist us writing something ourselves if something already exists
20:37:45 <heffer> <adamw> always trying to have at least two people doing something
20:37:50 <ianweller> <mmcgrath> doctors do it, plumbers do it, they develop their own way out of the problem
20:38:21 <ianweller> <J5> from a developer standpoint, if you're too rigid, you can only see with what you have, not the future
20:38:58 <ianweller> sometimes limiting yourself is good for creativity, but if you're only working with TG1... people got the idea of what we were designing for, but they didn't hit the mark, which explained how TG2 happened
20:39:08 <ianweller> those components exist because people were trying to solve the same problem
20:39:24 <ianweller> <skvidal> that's great, and it's wonderful if your task as a developer is solely and only "i produced this version of code and it works and walk away"
20:39:57 <ianweller> if you want to expand your creativity, say to yourself "i have to stick by this code for 5 years and if i go away i will be *gunned down*"
20:40:04 <ianweller> mmcgrath notes that skvidal wrote yum
20:40:21 <ianweller> <skvidal> after the tarball is released of whatever program, that's not enough, especially for webapps
20:40:27 <ianweller> <mmcgrath> maintenance costs are always huge
20:40:43 <ianweller> <skvidal> in the best of all possible worlds, TG2 would never exist. it'd be TG1, and they'd support it the whole way
20:40:51 <ianweller> and you wouldn't get to 2 for SEVERAL years
20:41:10 <ianweller> <mmcgrath> my sense is -- i'll defer if you wanna correct -- TG1 could've been compared to a beta, compared to TG2
20:41:27 <ianweller> <lmacken> 1 was built on what was the best at the time though
20:41:43 <ianweller> <toshio> TG1 was a beta, depending on how you look at it -- it was stable, but SQLAlchemy came out before TG1 came out
20:42:01 <ianweller> there was an internal debate in TG about whether to switch to SQLAlchemy or stay with SQLObject
20:42:16 <ianweller> cherrypy and sqlobject are now dead
20:42:42 <ianweller> <toshio> ideally we would be running TG1 for years!
20:43:21 <ianweller> <skvidal> let's say RHEL 6 -- seven years, right? -- ships with TG2
20:43:25 <ianweller> <skvidal> let's say it ships tomorrow!
20:43:37 <ianweller> are you ready for 2016 TG2 still being what you have to maintain?
20:43:52 <ianweller> <J5> i think you're trying to put the same sort of box on something that's a new technology -- if we thought that way, we'd use per
20:43:55 <ianweller> perl*
20:44:02 <ianweller> <mdomsch> nobody would have picked up python
20:44:11 <ianweller> <J5> so you have to say i know we're going to have headaches here, but the benefits are worth moving to it
20:44:21 <ianweller> <skvidal> that's true provided you have the same experience
20:44:42 <ianweller> if you're breaking API every 6 months, your users can't keep up
20:44:53 <ianweller> now what? they've all decided "we have to maintain compatibility"
20:45:16 <ianweller> look at yum and rpm -- we have to stay with 3.x compat for a few more years
20:45:40 <ianweller> you have to maintain those and you have to be prepared to maintain them for such a long period of time
20:45:54 <ianweller> it's a rapid development to a release -- by the time it's adopted by users, everything is obsolete
20:46:09 <ianweller> <mdomsch> same reason we don't run fedora on infra servers!
20:46:33 <ianweller> <smooge> the thing here though is that it's a constant -- don't want to use battle -- it's risk identification and putting two people in a room with a brick and one walks out
20:46:53 <ianweller> <jds2001> talking with toshio about this two days ago -- i don't like running the newest stuff because i have to fix it at 3am
20:47:04 <ianweller> <mmcgrath> the longer newer stuff has been around the better i can find it on google
20:47:27 <ianweller> there's a natural tendency for projects in themselves - we have TG2 devs - the project itself is realizing that they needed to release early/often and now that they're established they need to stagger that out
20:47:44 <ianweller> python frameworks established themselves even while it's an interesting time for python
20:48:28 <ianweller> <skvidal> python tried to find a way to match ruby on rails
20:49:17 <ianweller> [[ discussion on pylons, paste, TG, django, etc ]]
20:50:47 <ianweller> [[ now discussing deprecation cycles ]]
20:51:00 <ianweller> <jds2001> another question i have -- you keep mentioning versions -- what's the importance?
20:51:05 <ianweller> #topic why do versions matter
20:51:21 <ianweller> <mmcgrath> it's a completely abstract problem, and we can throw problems out there
20:51:49 <ianweller> "python 2.4 has been a prolbem" -- i can talk with these guys about it and it's all good but 2.4 alone is just a number
20:51:54 <ianweller> that 2.4 is nothing like turbogears 1 and 2
20:52:02 <ianweller> when you're talking about versions like this you really do have to have a context behind it
20:52:23 <ianweller> <j5> you'd hope that developers put meaning behind those numbers in some way.
20:52:44 <ianweller> TG1 - TG2 example
20:52:51 <ianweller> <lmacken> the moral of the story: the importance of the upstream!
20:53:17 <mchua> ianweller: I can spell you off for the remainder if you want a break
20:53:21 <ianweller> mchua: plz od
20:53:23 <ianweller> do*
20:53:41 <mchua> <seth> your best scenario is thousands of people using your stuff.
20:54:05 <mchua> <smooge> things are least oriented towards having other people support it long-term; they'll probably have another job by then.
20:54:32 <mchua> <j5> as projects mature, they need more stability - they can't keep implementing nifty new things.
20:54:42 <mchua> <mdomsch> everything the infra team is doing is ultimately in support of fedora devs and users.
20:54:56 <mchua> all the apps we've written have been directly in support for this.
20:55:05 <mchua> (Talking about custom maintained-by-Fedora, written-by-Fedora apps)
20:56:30 <mchua> <various programming language war debates>
20:57:03 <mchua> <jokes about age and who's been around for a loooong time>
20:57:12 <mchua> <adamw> I want my calendar system.
20:57:30 <mchua> <general good-natured teasing banter>
20:57:38 <mchua> <WAY TOO MANY OVERLAPPING VOICES TO LOG>
20:57:55 <mchua> ianweller: I can't lipread the question now
20:57:58 <mchua> ianweller: halp?
21:01:39 <mchua> ianweller: #endmeeting plz and post to https://fedoraproject.org/wiki/FUDCon:Toronto_2009_BarCamp_Schedule
21:03:54 <pcalarco> #meetingstart
21:04:08 <pcalarco> #meetingtopic Fedora and OLPC
21:04:39 <nirik> .addchair nirik
21:04:39 <zodbot> nirik: (addchair <channel> <network> <nick>) -- Add a nick as a chair to the meeting.
21:04:50 <nirik> .addchair #fudcon-room-7 nirik
21:04:50 <zodbot> nirik: (addchair <channel> <network> <nick>) -- Add a nick as a chair to the meeting.
21:04:55 <nirik> .addchair #fudcon-room-7 freenode nirik
21:04:55 <zodbot> nirik: Chair added: nirik on (#fudcon-room-7, freenode).
21:05:00 <nirik> #endmeeting