16:34:27 <dgilmore> #startmeeting RELENG (2015-02-23)
16:34:27 <zodbot> Meeting started Mon Feb 23 16:34:27 2015 UTC.  The chair is dgilmore. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:34:27 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:34:34 <dgilmore> #meetingname releng
16:34:34 <zodbot> The meeting name has been set to 'releng'
16:34:34 <dgilmore> #chair dgilmore nirik tyll sharkcz bochecha masta pbrobinson
16:34:34 <zodbot> Current chairs: bochecha dgilmore masta nirik pbrobinson sharkcz tyll
16:34:35 <dgilmore> #topic init process
16:34:39 <nirik> morning
16:34:45 * masta is here
16:34:50 * sharkcz is here
16:35:52 <dgilmore> #chair pingou
16:35:52 <zodbot> Current chairs: bochecha dgilmore masta nirik pbrobinson pingou sharkcz tyll
16:36:26 <pingou> o/
16:36:44 * jsmith lurks
16:36:48 <dgilmore> #topic FAD
16:37:02 <dgilmore> not sure who saw Pauls email last week
16:37:04 <dgilmore> but
16:37:07 <dgilmore> https://fedoraproject.org/wiki/FAD_Release_Tools_and_Infrastructure_2015
16:37:16 <dgilmore> please be sure to sign up for it
16:37:25 <pingou> https://lists.fedoraproject.org/pipermail/rel-eng/2015-February/019355.html
16:37:48 <dgilmore> there really is a massive amount of long term work here
16:37:53 <nirik> yeah, I have a tab with it open to look at. ;)
16:38:11 <dgilmore> I have forwarded the invite onto the internal Red HAt releng list to invite them all
16:38:28 <dgilmore> there is probably some infra folks that should be involved also
16:38:41 <pingou> be careful that not everyone shows up ;-)
16:39:01 <dgilmore> pingou: if we get work done and its productive I do not mind
16:39:53 <pingou> dgilmore: I agree with you
16:40:06 <pingou> just scared that it might not be the case :)
16:40:15 <dgilmore> indeed
16:40:18 <dgilmore> #topic #5931 [Proposal] Move new branch and unretire requests to pkgdb2
16:40:25 <dgilmore> https://fedorahosted.org/rel-eng/ticket/5931
16:40:39 <dgilmore> pingou: so I think this is almost complete now right?
16:40:47 <pingou> well, testing is still welcome, there has been a bunch of feedback received last week (thanks to Vít!)
16:40:59 <pingou> I'm sure it could use more testing on the admin side
16:41:09 <pingou> but otherwise, apparently it starts to look good yes :)
16:41:21 <nirik> so, land this after alpha is out?
16:41:26 <pingou> I guess we could push to prod after freeze
16:41:29 <pingou> yes :)
16:41:41 <pingou> maybe we should announce it a little more before?
16:41:49 <dgilmore> pingou: today or post freeze
16:42:16 <pingou> today is gonna be to soon, I expect we'll want to roll out some bug fix releases after the main one is done
16:42:19 <dgilmore> pingou: so perhaps send an email to devel-announce@  and ask for more testing
16:42:23 <pingou> +1
16:43:14 <nirik> sounds good.
16:44:05 <dgilmore> #topic #6016 Use fedpkg-minimal in Fedora buildroots
16:44:11 <dgilmore> https://fedorahosted.org/rel-eng/ticket/6016
16:44:26 <dgilmore> so we switched f22 and f23
16:44:50 <dgilmore> buildroot is smaller by a bit and takes about 20 seconds less to make a srpm on a x86 vm
16:45:01 <dgilmore> so we should get them all switched  now
16:45:27 <dgilmore> especially on epel5 where there is dep issues with some of teh surrounding tools and its not really supported
16:45:42 <dgilmore> #topic #5959 Enable keep-alive connections for koji (primary and secondaries)
16:45:47 <dgilmore> https://fedorahosted.org/rel-eng/ticket/5959
16:45:54 <dgilmore> I think this we just need to do
16:46:10 <dgilmore> #action dgilmore to implement this everywhere today
16:46:40 <dgilmore> #topic #5963 Orphaned vulnerable packages in EPEL
16:46:46 <dgilmore> https://fedorahosted.org/rel-eng/ticket/5963
16:47:01 <dgilmore> tyll: where is this at?
16:47:07 * nirik is +1 on the keepalive thing.
16:47:33 * pbrobinson is here
16:48:07 <dgilmore> guess tyll is not here yet
16:48:11 <dgilmore> #topic #6027 secondary arch old mash trees cleanup
16:48:16 <dgilmore> https://fedorahosted.org/rel-eng/ticket/6027
16:48:21 <dgilmore> pbrobinson: any update?
16:48:31 <pbrobinson> it's on my todo list for this week now I'm back to feeling human!
16:48:47 <dgilmore> feeling human for the win
16:49:23 <nirik> :)
16:50:09 <dgilmore> #topic #6108 Migrate to well-known CA for pkgs.fedoraproject.org
16:50:15 <dgilmore> https://fedorahosted.org/rel-eng/ticket/6108
16:50:24 <pbrobinson> nirik you can stop smiling.... it means I'll be chasing you're assistance for builders building ;-)
16:50:34 <dgilmore> this needs to wait until we do koji at the same time
16:50:44 <nirik> pbrobinson: Oh noe!
16:51:10 <dgilmore> it is going to effect end users
16:51:33 <nirik> yeah, we need to be carefull how we do it.
16:51:45 <dgilmore> we will need to make changes to the user configs to switch koji to using a well known ca for its https cert
16:52:06 <dgilmore> I would rather just do all the user effecting things once
16:52:21 <dgilmore> we do need to make a plan but it needs to be all encompasing
16:52:36 <nirik> sure. perhaps we can work on an exact plan in ticket/list?
16:52:41 <nirik> ie, steps that need doing
16:52:48 <dgilmore> the CA cert for koji infra expires in 2018, so we should do something at the same time with that also
16:52:56 <dgilmore> nirik: yeah
16:53:04 <pbrobinson> dgilmore: I presume you mean user affecting ;-)
16:53:12 <dgilmore> :P
16:53:21 <dgilmore> the easy bit is changing the server side
16:53:43 <dgilmore> but we need to coordinate changes on all developers systems at the same time
16:54:00 <pbrobinson> how does the possibly of using IPA for FAS3+ affect the timing?
16:54:14 <nirik> is it worth doing the source hash change at the same time?
16:54:27 <dgilmore> nirik: yes
16:54:53 <dgilmore> nirik: I think we should line up and do all the changes in that side of things at once
16:54:59 <nirik> pbrobinson: I don't think thats likely to happen... but we should discuss it some I suppose.
16:55:02 <dgilmore> with a big flag day
16:55:33 <dgilmore> pbrobinson: well it may come to play if we tie in dogtag as used by IPA for the user certs
16:56:34 <masta> this is seeming like a big project
16:56:42 <dgilmore> it is a big project
16:57:09 <dgilmore> mostly because there is a bunch of small things that should be coordinated because they all require user side changes
16:57:32 <dgilmore> it is not as simple as changing the cert used
16:58:30 <masta> so we have one phase where the user side is updated ahead of the next phase, the backend changes?
16:58:44 <masta> or do everythign at once?
16:59:00 <dgilmore> well we should do it all at once
16:59:17 <dgilmore> maytimes users run old versions of fedpkg and koji
16:59:20 <dgilmore> many times
16:59:36 <masta> hum... yeah, did not consider that.
16:59:43 <dgilmore> and they will need to have co-ordinated changes
17:00:04 <dgilmore> if we do it in steps they will need to update multiple times for the different phases
17:00:46 <dgilmore> we should just break it all at once. make the changes and tell people they have to get the updated versions to continue to work
17:01:06 <dgilmore> there really is not a good way to transition things
17:01:17 <dgilmore> because the clients verify the ca cert
17:01:26 <dgilmore> of the koji hub
17:01:38 <dgilmore> we would need to set up alternate names
17:01:48 <dgilmore> and it would likely cause a lot of confussion
17:01:51 <masta> yeah, we really do need to somehow make the older fedpkg break somehow, to force the update. Seem harsh, but understandably so
17:03:12 <dgilmore> well the change will break old versions of fedpkg and old koji configs
17:04:01 <dgilmore> there is just not a simple transparent way to transition things over
17:04:40 <pbrobinson> just have to have a flag day and have it well announced and well prepared
17:05:07 <dgilmore> maybe we could setup koji and pkgs on multiple ips and hijak /etc/hosts but that is nasty
17:05:15 <dgilmore> pbrobinson: yep
17:05:24 <nirik> +1 for flag day
17:05:49 <dgilmore> so for this ticket I will remove the meeting keyword, keep it open and retarget it for all the things we need to change
17:06:06 <pbrobinson> well you could use SNI and various other smart things but ultimately those hacks should only be around a short time and most of the time by the time they work it's easier to have a flag day and be done with it
17:06:51 <dgilmore> #action dgilmore to update ticket to reflect all changes needed for all changes in pkgs and koji setup, with a flag day at the end
17:07:08 <dgilmore> pbrobinson: right and that is probably much more effort and more drawn out
17:07:21 <dgilmore> #topic #6110 allow sudo for sysadmin-releng on sign bridges
17:07:28 <dgilmore> https://fedorahosted.org/rel-eng/ticket/6110
17:07:41 <dgilmore> I do not have a huge concern with this
17:08:10 <dgilmore> it would mean a extra number of people can access the box and restart things
17:08:39 <dgilmore> it is an extra attack vector but I think the people that would gain access already have access to fedora keys
17:08:55 <dgilmore> so have other ways to do bad things
17:09:00 <nirik> 2fa should be helpfull here too.
17:09:20 <masta> oh nice
17:10:08 <dgilmore> nirik: yeah, it will have to have 2fa
17:10:11 <tyll> I am here now
17:10:18 <nirik> it already is. ;)
17:10:28 <nirik> (well, on primary. Need to fix secondary when I reinstall it)
17:10:42 <masta> I'd like to have sign-bridge self-service =)
17:11:04 <masta> one less thing I have to ask nirik to help with ;-)
17:11:12 <dgilmore> masta: I would rather have it not need to be serviced often
17:11:19 <pbrobinson> masta: then what would you actually do?
17:11:22 * nirik nods to both
17:11:38 <masta> dgilmore: yeah, ideally it wouldn't need service, that needs looking at
17:12:08 <masta> pbrobinson: hehe... well all I do it sign & push, so... that
17:13:27 <dgilmore> okay
17:13:54 <dgilmore> #topic #5963 Orphaned vulnerable packages in EPEL
17:14:00 <dgilmore> https://fedorahosted.org/rel-eng/ticket/5963
17:14:08 <dgilmore> tyll: what is the current status?
17:15:57 <tyll> I am waiting for fedpkg-minimal to be used in EPEL
17:16:00 <tyll> EPEL5
17:16:05 <tyll> especially
17:16:29 <dgilmore> okay
17:17:40 <dgilmore> #topic Secondary Architectures updates
17:17:41 <dgilmore> #topic Secondary Architectures update - ppc
17:17:51 <dgilmore> pbrobinson: sharkcz: how is ppc world?
17:18:47 <pbrobinson> getting there, sharkcz is back this week so he might be able to provide a better update
17:19:15 <sharkcz> relatively good, but we are seeing a broken code produced by gcc5 on ppc64 and s390, but is created, working on reduced test case for Jakub
17:19:31 <dgilmore> okay :(
17:19:32 <sharkcz> e2fsprogs exposes the bug
17:19:54 <dgilmore> #topic Secondary Architectures update - s390
17:20:00 <pbrobinson> I've seen a number of issues across all architectures
17:20:09 <sharkcz> as noted above
17:20:11 <dgilmore> yeah
17:20:19 <dgilmore> #topic Secondary Architectures update - arm
17:20:19 <pbrobinson> we're certainly not ready for a mass rebuild with it yet
17:20:25 <pbrobinson> we're getting up to date
17:20:34 <pbrobinson> fixed up a few issues over the weekend
17:20:38 <dgilmore> cool
17:20:47 <pbrobinson> and we should now catch up over the next day or so
17:20:54 <dgilmore> excellent
17:21:01 <pbrobinson> biggest issues is currently ghc and I'm hoping that gets fixed RSN
17:21:09 <dgilmore> has gcc5 been mostly okay for aarch64
17:21:16 <dgilmore> at least about the same as x86_64
17:21:27 <nirik> we have now installed the new s390 box... we set it up as a virthost so we can run a hub/db in vm's...
17:21:38 <dgilmore> cool
17:21:50 <sharkcz> nice
17:21:52 <dgilmore> nirik: so we will need to ansible the vms?
17:21:55 <nirik> I'd like to work with pbrobinson and sharkcz on whichever you like of ppc/s390/arm to setup a new hub
17:22:19 <nirik> yeah. I figure we setup a new hub/db for one of them, get it working then cut over to it.
17:22:32 <dgilmore> nirik: s390 and arm have builder roles on the hub
17:22:48 <nirik> I'm sure our primary hub playbooks will need adjustments to be more generic.
17:22:55 <nirik> ok
17:23:00 <dgilmore> they will need quite a bit I think
17:23:11 <nirik> koji01.stg has a builder setup also on it.
17:23:21 <pbrobinson> dgilmore nirik: although I think the arm builder options on hub could go once we get the local mustangs rebuilt?
17:23:32 <dgilmore> pbrobinson: they could
17:23:33 <nirik> pbrobinson: yeah, it was mostly for newrepos...
17:23:42 <dgilmore> nirik: its only for newRepos
17:23:44 <sharkcz> nirik: I'll provide you a setup for shared shadow configs
17:24:07 <sharkcz> for multiple people on the hub
17:24:09 <dgilmore> nirik: did we setup sysadmin-secondary?
17:24:20 <nirik> dgilmore: not yet, we can...
17:24:24 <nirik> sharkcz: cool
17:24:42 <nirik> dgilmore: what would that get access to? all the secondary hub/db'svirthosts?
17:24:47 <dgilmore> nirik: either a secondary only group or use sysadmin-releng
17:24:55 <dgilmore> nirik: yeah
17:25:09 <nirik> either is fine with me. Are all the secondary folks in sysadmin-releng?
17:25:30 <dgilmore> nirik: I think we may want both
17:25:32 <pbrobinson> I think probably a -secondary group would be useful
17:25:37 <dgilmore> ssyadmin-releng with sudo
17:25:39 <sharkcz> we will need a group for regular shadow users too
17:25:50 <dgilmore> and sysadmin-secondary with access but no sudo
17:26:04 <sharkcz> I can explain it later
17:26:23 <nirik> dgilmore: ok. can set it up that way.
17:26:44 <dgilmore> sharkcz: please do. shadow should only be run by one account
17:27:05 <dgilmore> sharkcz: longer term i plan to make shadow run as a part of internal fedora infra
17:27:09 <sharkcz> dgilmore: correct, I'll send an email to releng list tomorrow
17:27:22 <dgilmore> sharkcz: okay
17:27:56 <dgilmore> nirik: I guess we should wait to see what sharkcz is going to propose
17:28:03 <nirik> k
17:28:17 <nirik> I don't want to rush things, but keep moving them along.
17:28:22 <dgilmore> yep
17:28:31 <dgilmore> I want to get things moving along and consisten
17:28:32 <nirik> we will be in a much better place once all the secondaries are in ansible/etc.
17:28:34 <dgilmore> consistent
17:29:04 <dgilmore> nirik: I guess short term we could allow only sysadmin-releng with sudo access
17:29:14 <dgilmore> then we can change it down the road
17:29:18 <nirik> sure.
17:29:39 <dgilmore> anything else secondary arch related?
17:29:58 <pbrobinson> not from me
17:29:59 <sharkcz> storage, but we can discuss it offline
17:30:08 <dgilmore> okay
17:30:12 <dgilmore> #topic Open Floor
17:30:13 <nirik> sharkcz: yeah. ;( we are still waiting for new netapp stuff.
17:30:16 <nirik> it's very... slow
17:30:20 <sharkcz> what are the options, should we plan for next FY, etc
17:30:24 <dgilmore> tomorrow is change freeze for F22 alpha
17:30:36 <tyll> I have some comments for #6108 Migrate to well-known CA for pkgs.fedoraproject.org
17:30:36 <nirik> sharkcz: we have space on new netapps as soon as they are setup and we can migrate to them
17:30:45 <dgilmore> tyll: okay?
17:30:49 <nirik> sharkcz: if they would ever finish setting them up. ;(
17:31:29 <tyll> There is already an outline of actions that is needed for pkgs at https://fedorahosted.org/fedora-infrastructure/ticket/2324#comment:8
17:31:29 <sharkcz> nirik: ok, one would think it is now a plug and play technology ... :-(
17:31:35 <dgilmore> tyll: it is not doable
17:31:44 <nirik> sharkcz: these new ones are c-mode (some new thing)
17:31:46 <dgilmore> tyll: it requires user side changes
17:32:00 <dgilmore> tyll: and we know we have other things requiring user side changes
17:32:07 <dgilmore> tyll: we need to batch them into one
17:32:18 <dgilmore> I told you that last week in #fedora-releng
17:33:01 <dgilmore> I said we can move it earlier only if it did not require user side changes
17:33:55 <tyll> Not sure, which part is not doable, we can already start with Step 1 and make fedpkg ready for the change right now and then do the switch at pkgs whenever we want
17:34:32 <dgilmore> tyll: we can not change the cert on the server
17:34:59 <dgilmore> some people still use very old versions of the tools
17:35:29 <dgilmore> we have a bunch of changes we need to make that will require updated tooling on the client side
17:35:53 <dgilmore> we need to make them all at once. it is going to break things, so we will have a flag day for the changes
17:36:10 <tyll> yes, the first change is to make fedpkg ready for changing the cert on the server, but this does not require to actually change the cert already
17:36:40 <tyll> e.g. if we update fedpkg now, we can still switch the cert in a month or whenever we do the hashing update for sources
17:36:45 <dgilmore> tyll: we can not do any step after step 1
17:36:55 <dgilmore> tyll: we can not
17:37:11 <tyll> We can make fedpkg accept both the current and the future certificate
17:37:27 <dgilmore> tyll: we will update the cert the same time we change koji over and change the hashing and all the other things we are changing in that space
17:38:07 <dgilmore> tyll: we will not change the cert on the server until we do koji at the same time. we can land changes to the clients that do not break anything
17:38:16 <dgilmore> and I am not saying we can not do that
17:38:31 <dgilmore> some of the client side changes will not work with the existings etup
17:39:36 <tyll> dgilmore: ok, so is it ok to update staging pkgs to the proper certificate and make fedpkg work with both the staging setup and the real one?
17:39:49 <nirik> dgilmore: I had a unrelated open floor issue...
17:39:57 <nirik> (after this discussion)
17:40:02 <dgilmore> tyll: I would rather not until we are working on changing everything
17:41:11 <tyll> dgilmore: for koji we only need to switch the cert and remove the old client CA check code, I submitted a patch for this and the hash changing is also already pretty much evolved afaik
17:41:24 <dgilmore> tyll: its not that simple
17:41:53 <tyll> dgilmore: the patch keeps the API
17:41:55 <dgilmore> tyll: we need to consider scheduling,
17:42:14 <dgilmore> along with planninng the client side tooling changes
17:42:32 <dgilmore> we need to do something with the CA cert we currently use as it expires in 2018
17:42:57 <dgilmore> tyll: there is a very complex set of things we need to look at when making the plan for the change
17:43:12 <dgilmore> we will not be able to do it before Fedora 22 is GA
17:44:07 <tyll> What kind of things need to be done with the CA cert that expires?
17:44:47 <tyll> And what are the open questions for the client side tooling that need to be planned?
17:44:48 <dgilmore> tyll: that is the CA that signs all the user certs and builder certs
17:45:04 <dgilmore> its what signs the cert currently used by koji and pkgs
17:45:17 <dgilmore> right now it is a spof
17:45:26 <tyll> yes, but the client CA part does not depend on this
17:45:31 <dgilmore> you can only make user certs on one of the fas backends
17:46:10 <dgilmore> tyll: on the user side we need to change the koji configs.
17:46:20 <tyll> e.g. all the tools just need to get a new client cert when we switch the client CA in 2018, but this is needed for regular uses every 6 months
17:46:21 <dgilmore> tyll: we only really get one shot at doing so
17:46:37 <tyll> dgilmore: the user side does not need to know the client Ca
17:46:39 <dgilmore> because you can never be sure that people have updated the software
17:46:58 <dgilmore> tyll: it does because the cert is from it.
17:47:24 <dgilmore> tyll: I really wish I could make you understand how wide the change effects
17:47:44 <tyll> dgilmore: the client does not need to know the client CA just because its own certificate is signed by the client CA
17:47:47 <dgilmore> because your view seems to be so narrow that you do not see the side effects of the change
17:48:07 <dgilmore> tyll: I guess I am not explaining it well
17:49:49 <dgilmore> tyll: it boils down to this
17:49:49 <tyll> afaics we have three things that need to be done before 2018: 1) Switching the server cert on pkgs and update fedpkg for it 2) Switching the server cert on koji and update the koji client configs for this 3) Update the client CA and issue new client certs and update the server koji config
17:50:15 <tyll> 1) and 2) do not depend on 3) or vice-versa
17:50:19 <nirik> 4) change any tooling that requests new certs to use whatever we setup for a backend?
17:50:25 <dgilmore> we have a bunch of changes to make in koji, pkgs and related tooling.  we are going to make all teh changes at once and have a flag day
17:50:30 <dgilmore> discussion is finished
17:51:12 <dgilmore> tyll: we are doing it all at once
17:51:15 <tyll> so this is something that will happen with koji2?
17:51:24 <dgilmore> not necessarily
17:51:43 <dgilmore> but it is something that we are going to batch all teh changes into one event
17:52:13 <dgilmore> and it is not going to happen until after F22 is out at the earliest
17:52:28 <tyll> what are the other changes? It might make it more clear if we have tickets for all of them
17:52:38 <dgilmore> we need to make a plan and document all the things effected and changes that need to be made
17:53:06 <dgilmore> nirik: floor is yours
17:53:25 <nirik> just wanted to ask if we were still planning on moving to livemedia-creator in f22...
17:53:34 <dgilmore> nirik: not at the moment
17:53:47 <dgilmore> not had the time to integrate it into koji
17:53:56 <dgilmore> it really landed too late
17:54:04 <nirik> ok. we should really try then to enable rawhide for it
17:54:06 <nirik> not wait
17:54:22 <dgilmore> nirik: indeed, but we need koji code written to enable it
17:54:39 <masta> dgilmore: the runroot code?
17:54:43 <dgilmore> masta: no
17:55:34 <nirik> I thought bcl had a patch for that?
17:55:39 <nirik> or perhaps I am misremembering
17:55:51 <dgilmore> nirik: no he wants us to run it in mock outside of koji
17:56:05 <dgilmore> which is not something I want
17:56:20 <dgilmore> as it takes them completely away from being user visable
17:56:37 <dgilmore> and makes it that we have to run them in serial rather than take advantage of the builders
17:57:01 <nirik> sure. I thought that was solved, but ok
17:57:12 <dgilmore> nirik: if it is its news to me
17:57:38 <nirik> ok, will try and sort it out then.
17:58:40 <dgilmore> okay
17:58:45 <dgilmore> anything else?
17:59:39 <dgilmore> #endmeeting