18:00:19 <nirik> #startmeeting Infrastructure (2015-09-24)
18:00:19 <zodbot> Meeting started Thu Sep 24 18:00:19 2015 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:19 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:00:19 <nirik> #meetingname infrastructure
18:00:19 <nirik> #topic aloha
18:00:19 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk pbrobinson
18:00:19 <zodbot> The meeting name has been set to 'infrastructure'
18:00:19 <zodbot> Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pbrobinson pingou puiterwijk relrod smooge threebean
18:00:20 <nirik> #topic New folks introductions / Apprentice feedback
18:02:15 <nirik> any new folks like to give a short one line introduction? or apprentices with questions or comments?
18:02:46 * threebean is here
18:02:52 * aikidouke is here
18:02:55 <dgilmore> hi
18:03:09 * danofsatx is here, but no longer in the program :(
18:03:14 * pcreech|work is here
18:04:24 <nirik> ok, if no intros or apprentice questions, lets move on to status / info:
18:04:27 <nirik> #topic announcements and information
18:04:27 <nirik> #info Fedora 23 Beta is out the door smoothly - everyone
18:04:27 <nirik> #info Next freeze (final) coming up 2015-10-13 - kevin
18:04:27 <nirik> #info new release of the-new-hotness went out this morning (with some hiccups) - ralph
18:04:28 <nirik> #link https://lists.fedoraproject.org/archives/list/infrastructure%40lists.fedoraproject.org/thread/TXYVETKPKSD7DZER6DLJYIGGJKWFIVPF/
18:04:29 <nirik> #info our infrastructure git repos (ansible, puppet, dns, etc..) now publish fedmsg messages about commits - relrod/ralph
18:04:32 <nirik> #link https://apps.fedoraproject.org/datagrepper/raw?category=infragit&delta=127800
18:05:02 <nirik> I'd like to add a item: I have a outage scheduled for tomorrow at 16:00UTC to migrate lockbox01 to batcave01.
18:05:26 <nirik> I'll be swapping IP addresses, so ssh will tell you it's changed and you will need to update your known hosts.
18:05:53 <nirik> and I am sure I will have missed some things, because lockbox01 in puppet was many layers of stuff (Althought I tried to get it all)
18:06:06 * roshi is here as well
18:06:07 <nirik> but we can work through them after the migration.
18:06:16 <nirik> Note that /home will be synced.
18:06:44 <nirik> any other announcements or status?
18:06:59 * pingou late
18:07:02 <nirik> ok, moving along.
18:07:05 <nirik> #topic Grant admin permission in staging Koji to mizdebsk
18:07:05 <nirik> .ticket 4900
18:07:07 <zodbot> nirik: #4900 (Grant admin permission in staging Koji to mizdebsk) – Fedora Infrastructure - https://fedorahosted.org/fedora-infrastructure/ticket/4900
18:07:20 <mizdebsk> koschei testing in staging has been quite painful, mostly due to staging koji not working (which is fixed now, thanks to nirik and threebean) and due to lack admin permission to simulate corner cases
18:07:23 <nirik> mizdebsk: you here? thanks for listing things clearly there, that helps.
18:07:26 <mizdebsk> i asked for admin permission on releng trac about half year ago, but there hasn't been any progress, so i'm bringing this issue here in hope you'll help with resolving it
18:07:35 <mizdebsk> are there any reasons why i wouldn't be granted admin permission in *staging* koji only?
18:07:39 * lnxslck is here
18:07:57 <nirik> well, it wasn't too clear what you needed it for, I think last time we talked you said you needed to import src.rpms, which we don't want to do...
18:08:06 <nirik> but I think this ticket is more clear
18:08:23 <mizdebsk> this has been fixed now - i can build from staging dist-git, so no need to import srpms any longer
18:08:33 <nirik> I personally don't have any objections. dgilmore: can you look at the ticket and let us know what you think?
18:08:36 <nirik> yeah
18:09:15 <nirik> stg koji is much more usable than it used to be
18:09:33 <nirik> mostly due to threebean''s work on the prod->stg script
18:09:55 <mizdebsk> yes, definitely
18:10:15 <nirik> I''ve not yet done image builds on it, but we should make sure that works too
18:10:48 <mizdebsk> i will revert every change i make for purposes of testing koschei, but we have the playbook to resync stagig koji from prod in case something would go terribly wrong
18:10:58 <threebean> my two cents:  I think its reasonable to give out staging koji admin rights more broadly
18:11:23 <nirik> mizdebsk: idle thought: could koschei be extended to images? ie, if you know all packages that go into an image or are needed to make it, you could rebuild it anytime those things change?
18:11:26 <threebean> it doesn't apply in this case, but it would be nice to be a ground where we could develop new koji plugins (I know maxamillion was looking at this, and imcleod wanted to try stuff out too)
18:11:42 <nirik> I guess thats a sidetrack sorry
18:12:06 <mizdebsk> nirik: i don't know, haven't thought about images
18:12:40 <nirik> threebean: yeah.
18:12:54 <nirik> I want to try and test new mock in stg before we roll to prod.
18:13:03 <nirik> and rpm and dnf, etc
18:13:50 <pingou> test critpath in stg before we roll to prod? :)
18:14:16 <nirik> sure. Mostly for things like changes to the builders.
18:14:23 <pingou> +1
18:14:25 <nirik> when we move them to f23 we should test that in stg too
18:15:13 <dgilmore> I would rather we dont give admin to stg koji in order to sidstep doing things that are supposed to be doen elsewhere
18:15:24 <pingou> dgilmore: as in?
18:15:27 <pingou> done where?
18:15:40 <dgilmore> so for koschei we should let mizdebsk be able to approve packages etc in pkgdb
18:15:57 <dgilmore> that way we test the full suite of interrelated apps
18:16:08 <nirik> sure, but he may need admin for the other items listed in the ticket.
18:16:24 <nirik> like canceling stuck jobs or changing something to see how it breaks kschei
18:16:34 <mizdebsk> what about blocking? am i supposed to wait half a day to test a code path? blocking directly is much simpler
18:17:14 <nirik> additionally, any changes made in stg will be completely wiped out when the next sync happens.
18:18:10 <nirik> being able to run repo-regen is also important
18:19:03 <dgilmore> you do not need admin to cancel stuck jobs
18:19:28 <nirik> well, admin or the koschei cert.
18:19:49 <mizdebsk> i can simulate regen-repo with untag/tag on any random build, but then kojira generates both f24-build and rawhide, which is slow - only one host in createrepo channel
18:20:12 <dgilmore> anyway, if there is valid reasons outside of adding and removing packages then we can look at it
18:20:15 <nirik> yeah, I can add more, thats due to them not havig rw /mnt/koji
18:20:31 <dgilmore> I would just rathe we try to follow production process
18:21:07 <pingou> if we can't test things in stg what is it really good for? Do we need a thrid environment, test ?
18:21:09 <nirik> sure, thats why we have been fixing up stg to behave like prod
18:21:39 <dgilmore> pingou: I am not sure I follow you there
18:21:42 <mizdebsk> i'll be following prod process whenever possible, but in many cases it is very impractical or impossible and would mean not testing certain features or code paths at all
18:22:03 <pingou> dgilmore: I don't quite see the harm in letting people do changes in stg to test other application and giving them the rights to do so
18:22:32 <dgilmore> pingou: still not following you
18:22:33 <pingou> we're already having troubles to replicate prod in stg and if we can't make stg such that it can be use for integration testing
18:23:00 <pingou> then I wonder what we need to do to have an environment we can test in
18:23:52 <dgilmore> pingou: if we can make mizdebsk have admin rights in pkgdb he can add and remove packages as needed
18:24:02 <pingou> in stg that's an easyfix
18:24:14 <dgilmore> it hads the benefit of testing the full suite of interconnected apps
18:24:24 <pingou> if it's all what mizdebsk needs it'll be done in a few minutes :)
18:24:29 <dgilmore> rathe than just adding the package to koji and building
18:24:45 <nirik> but the problem with treating stg as just another happily functioning prod is that you then ignore the corner cases that could hit prod
18:24:49 <dgilmore> thats the kind of thing I am trying to avoid
18:24:50 <mizdebsk> pingou: no, that definitely doesn't satisfy my needs for thesting koschei
18:25:11 <pingou> mizdebsk: that was also my impression :)
18:25:17 <nirik> so IMHO it's valid and valuable to be able to introduce errors in stg and confirm dependent apps still handle things
18:25:27 <dgilmore> my impression was that is all thats needed
18:25:36 <dgilmore> so apparently I have missed something
18:25:36 <nirik> especially when you can just wipe and sync to prod anytime to remove whatever you did
18:26:34 <nirik> https://fedorahosted.org/fedora-infrastructure/ticket/4900 has a list
18:27:27 <dgilmore> I will have to read the ticket later
18:27:34 <dgilmore> but I have not yet seen it before now
18:28:20 <nirik> well, it would be nice to approve this, since he's been waiting a long time...
18:28:24 <pingou> dgilmore: out of curiosity, what makes you relunctant in giving access to koji in stg?
18:28:40 <nirik> IMHO we should also be free to grant it to the people working on plugins.
18:28:51 <pingou> is there a security concern or an aspect we're not thinking about?
18:30:02 <dgilmore> pingou: I don't want things to be done in a way where there becomes a reliance on it in prod
18:30:39 <nirik> I don't see how that would be possible here...
18:30:49 * pingou has an english failure there
18:31:35 <nirik> since mizdebsk won't be admin in prod no changes would be made there... and if he made changes in stg that koschei depended on it would break when the next prod->stg syc happened or when that version went to prod
18:31:38 <pingou> we rely on something to be working in a specific way in prod because we adjust stg to behave that way?
18:32:40 <pingou> and that would be koschei's problem (thus mizdebsk) koji wouldn't be impacted
18:34:04 <dgilmore> I am failing to grok what you are trying to say pingou
18:35:07 <pingou> the 'we rely on something'  was me trying to rephrase what you said to see if I understood it correctly :)
18:35:21 <pingou> the 'and that would be...' was a follow up on nirik's sentence
18:36:02 * nirik is not seeing the big deal here. If it was prod sure, but this is just stg, it's specifically for testing applications before they go to prod
18:36:11 <dgilmore> I have two main concerns, 1) is that I do not feel that admin should ever be needed for koschei to do its thing, 2) that we will skip the testing of the other pieces in the packaging puzzle because it is easier to do
18:36:51 <pingou> for 1) iiuc it's not for koschei to do things it's for mizdebsk to test koschei properly
18:37:09 <pingou> for 2) I'd say that this is mizdebsk's problem :)
18:37:09 <nirik> 1) agreed in prod, but not in in stg, in stg we want to be able to change things to break koschei so it's properly tested.
18:37:47 <mizdebsk> ad 1) koschei user in koji will not have admin rights, only me - koschei admin
18:38:18 <mizdebsk> ad 2) i won't be waiting half a day to test package blocking, so this wouldn't be tested anyways
18:39:12 <threebean> put another way, mizdebsk needs to be able to break koji.
18:39:42 <lmacken> since there are conflicting opinions, shall we vote on it?
18:40:00 <mizdebsk> yes, sort of - temporarily to test how koschei reacts to corner cases; and then revert changes asap (un-break it)
18:40:03 <dgilmore> I am going to step aside because it seems no one cares what my concerns are
18:40:34 <nirik> I do... I just don't understand them. ;(
18:40:45 * lmacken doesn't understand dgilmore's #2
18:41:00 <dgilmore> lmacken: testing pkgdb, etc
18:41:10 <nirik> I kinda get that one...
18:41:17 <lmacken> we have rube for testing as many apps in stg as possible, and the pkgdb is in there.
18:41:26 <nirik> if we block a package in pkgdb.stg it might somehow behave differently from a manual koji admin block
18:41:56 <nirik> and then if we test via the manual thing it break in prod.
18:42:13 <mizdebsk> blocking doesn't happen immediately - only after very long delay - at which point i'll be sleeping and not able to observe koschei behaviour
18:42:13 <dgilmore> that shouldn't happen
18:42:18 <nirik> I understand that danger, but I also think it's still worthwhile to test it other ways.
18:43:24 <mizdebsk> some cases are not possible to simulate with pkgdb, like removing package (not blocking)
18:43:43 <dgilmore> I would like for us to do more work on dist-git and make it more useful
18:43:44 <pingou> mizdebsk: pkgdb can remove packages
18:43:48 <nirik> also, it should be more on mizdebsk... if he tests in a way that doesn't reflect how it's done in prod and causes a bug to get out hes the one that has to fix it and know to improve the way it's tested.
18:43:51 <pingou> just have no idea of its effect in koji
18:44:09 <mizdebsk> pingou: i meant removing package from koji tag listing, not from pkgdb
18:44:24 <pingou> k
18:44:29 <dgilmore> mizdebsk: we remove a package by blocking
18:44:47 <dgilmore> that is how it is done
18:45:11 <dgilmore> we do not delete a package from existance
18:45:13 <mizdebsk> usually, but not always; sometimes package gets removed and this happened in the last month
18:45:29 <nirik> there have been cases due to legal
18:45:44 <dgilmore> which afaik has only happened once
18:45:44 <mizdebsk> this happens eg. when stg koji is sysced with prod; if you added pkg to staging only it would be gone
18:45:58 <dgilmore> mizdebsk: we do not do that in prod though
18:46:10 <dgilmore> except is legal says it need stobe gone
18:46:18 <dgilmore> which we did once years ago
18:46:21 <nirik> yeah, I can only recall that once....
18:46:24 <mizdebsk> and i would like to test that case too
18:46:46 <dgilmore> mizdebsk: to remove it like that you have to go into the db
18:46:54 <dgilmore> being a koji admin it is not possible
18:47:13 <mizdebsk> for my purposes it's enough to remove from tag listing, which can be done from xml-rpc
18:47:21 <dgilmore> you need to manually crate a delete query
18:49:50 <nirik> ok, so we have been on this a while, not sure what we want to do. dgilmore: how about we provisionally add him and revisit next week when we can look at koji.stg history?
18:50:17 <nirik> then if there's things we have more direct concerns about we can bring them up then?
18:52:13 <dgilmore> ehh
18:53:09 <nirik> well, that wasn't a no. ;)
18:53:13 <nirik> anyhow we are running out of time...
18:53:14 <pingou> ^^
18:53:18 <nirik> #topic Open Floor
18:53:23 <nirik> any items for open floor?
18:54:02 <smooge> not from me
18:54:10 <pingou> dinner's ready :)
18:55:01 * nirik had something, but now cannot recall what it was.
18:55:37 <pingou> batcave01?
18:56:05 <kalev> I had a question I was going to ask nirik, but might as well do it here
18:56:18 <nirik> pingou: already mentioned early in the meeting. ;)
18:56:23 <nirik> kalev: sue
18:56:24 <nirik> sure
18:56:31 <kalev> my question was about https://fedorahosted.org/fedora-infrastructure/ticket/4857
18:56:58 <kalev> does anyone have ideas how to make private builds happen in koji, where the binaries produced wouldn't be downloadable for general public?
18:57:08 <nirik> well, lets take this discussion off line
18:57:11 <kalev> ok
18:57:14 <nirik> this does not need to be in the infra meeting.
18:57:43 <dgilmore> kalev: thats a massive massive undertaking
18:58:11 <dgilmore> kalev: you would be best off talking to mikem and mikeb but the answer is it is not simple
18:58:23 <dgilmore> but it is off topic for this meeting
18:58:51 <kalev> sure, I can definitely talk to them
18:58:54 <kalev> thanks dgilmore
18:59:01 * nirik was going to test a theory about this on stg koji... but hasn't had a chance yet
18:59:09 <nirik> anyhow, if nothing else will close out in a minute.
19:00:03 <nirik> thanks for coming everyone!
19:00:06 <nirik> #endmeeting