18:00:36 <relrod> #startmeeting Infrastructure (2017-09-28)
18:00:36 <zodbot> Meeting started Thu Sep 28 18:00:36 2017 UTC.  The chair is relrod. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:36 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:00:36 <zodbot> The meeting name has been set to 'infrastructure_(2017-09-28)'
18:00:36 <relrod> #meetingname infrastructure
18:00:36 <zodbot> The meeting name has been set to 'infrastructure'
18:00:36 <relrod> #topic aloha
18:00:36 <relrod> #chair smooge relrod nirik abadger1999 dgilmore threebean pingou puiterwijk pbrobinson maxamillion
18:00:36 <zodbot> Current chairs: abadger1999 dgilmore maxamillion nirik pbrobinson pingou puiterwijk relrod smooge threebean
18:00:48 <puiterwijk> Hai
18:00:50 <relrod> Hi people, happy Thursday
18:00:50 <nirik> morning
18:00:55 <jcline> Hello
18:00:59 <bt0> hi
18:01:02 <netcronin> .hello
18:01:02 <zodbot> netcronin: (hello <an alias, 1 argument>) -- Alias for "hellomynameis $1".
18:01:03 <bowlofeggs> .hello2
18:01:06 <zodbot> bowlofeggs: bowlofeggs 'Randy Barlow' <randy@electronsweatshop.com>
18:01:13 <puiterwijk> Happy Fedora 27 Beta GO day
18:01:20 <relrod> \o/
18:01:25 <bowlofeggs> nd33d
18:01:27 * cverna waves to the crowd
18:01:36 <bowlofeggs> the crowd waves back
18:02:16 <smooge> hello
18:02:28 <benniej> hi
18:02:47 <relrod> #topic New folks introductions
18:02:48 <relrod> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
18:03:00 * bowlofeggs offers smooge a warm la croix
18:03:01 <relrod> Any newcomers who would like to introduce themselves?
18:04:06 <relrod> Don't be shy
18:04:19 <benniej> My name Bennie. Live and work in Pretoria, South Africa. Interests are networking, systems administration and development/packaginh
18:04:35 <nirik> welcome benniej!
18:04:39 <cverna> welcome benniej
18:04:39 <relrod> cool, welcome!
18:04:41 <benniej> Thnx
18:04:43 <netcronin> welcome benniej
18:05:19 * threebean waves
18:06:00 <bt0> welcome
18:06:01 <relrod> benniej: hang around in #fedora-admin after the meeting and we can help you get started/get into the fi-apprentice group so you can start looking around.
18:06:19 <benniej> awesome thnx
18:06:51 <relrod> okay, moving on to announcements
18:06:56 <relrod> #topic announcements and information
18:06:56 <relrod> #info PHX2 Colo Trip, Dec 4th - 9th
18:06:56 <relrod> #info Infrastructure will be down during that trip.
18:06:56 <relrod> #info beta freeze still going on.
18:06:57 <relrod> #info Moving .stg. ips is ongoing this week - smooge
18:06:58 <relrod> #info .stg. vh 16 and vh20 moved over .. broke lots of things
18:06:59 <relrod> #info fedora-packages, looking at a way to replace xappy (dead upstream) - cverna
18:07:28 <nirik> smooge: so whats left for stg? vh11? or more?
18:07:40 <nirik> and cverna: whats the packages status currently? :)
18:08:04 <smooge> we have multiple boxes
18:08:52 <smooge> nirik, vh11 vh04
18:08:57 <smooge> about 30 guests
18:09:05 <smooge> and god knows how many hard coded ips
18:09:24 <nirik> ok. happy to help, and thanks for working on it
18:09:29 <cverna> To be able to get packages to rhel7 we need to get rid of xappy which is used by the indexer to populate teh xapian db
18:09:35 <relrod> happy to help as well
18:09:46 <netcronin> Same. If I get done with homework this weekend I'll hop on.
18:09:59 <netcronin> I just uploaded another patch to the ticket with the IPs I changed.
18:10:00 <smooge> the ones I have skipped on purpose at the moment is proxy01, koji01, pkgs01, and pgpdr
18:10:21 <nirik> cverna: yeah. ;( Another bug got filed today... the 'current releases' isn't updating.
18:10:21 <cverna> I will try to use plain xapian bindings since the rest of the app works with them.
18:10:38 <cverna> nirik: I have seen
18:11:12 * bowlofeggs switches bodhi's configs to use IP addresses for other services instead of domain names…
18:11:20 <cverna> I ll try to have some ready for unfreeze time
18:11:33 <smooge> nirik, relrod netcronin if you want to also stage the patches for the host_vars for vh04 that would be good. A master playbook then needs to be run on all of staging and then a test to make sure ansible did what it said it would do
18:11:56 <cverna> s/some/something working
18:12:27 <nirik> cverna: sounds great. Thanks for working on this. So likely it's not worth the el6 version in stg now? since we hopefully will have el7 soon?
18:12:37 <netcronin> smooge: Is vh04 an actual hostname?
18:12:42 <smooge> virthost04
18:12:44 <relrod> smooge: ok. grep the /etc/hosts files on all stg hosts for hostnames of all boxes changed, too.
18:12:46 <relrod> I think
18:12:54 <smooge> relrod, I am working on that
18:12:56 <netcronin> Okay, I'll try.
18:13:09 <smooge> that is my patch while netcronin does the host_vars
18:13:21 <relrod> smooge: ok :)
18:13:39 <smooge> there is also changes in postgres database files and changes in iptables that will need to be dealt with
18:13:57 <relrod> yep
18:14:04 <cverna> nirik: the latest packages will not work on el6 anymore :P , I have remove the python-appstream dependencies
18:14:05 <nirik> I can move pgbdr...
18:14:16 <nirik> cverna: ok, fair enough
18:15:00 <relrod> Okay, any other comments on the announcements, or things people would *like* to announce?
18:15:11 <smooge> we got a beta maybe?
18:15:27 <relrod> oh, yeah
18:15:31 * cverna will break his teeth on xapian over the week end :)
18:15:36 <relrod> #info Beta GO! woot!
18:15:44 <cverna> \o/
18:16:19 <relrod> alright
18:16:25 <relrod> #topic Moving apps into OpenShift
18:16:37 <nirik> I think this is leftover from last week... ;)
18:16:48 <nirik> no one has had cycles to work on it yet
18:17:15 <relrod> yeah. I'd like to start on some of our smaller apps, in stg (fedocal comes to mind. Possibly elections). I will start on that next week
18:18:00 <bowlofeggs> doing some small apps would be nice to help us develop some guidelines
18:18:06 * relrod nods
18:18:15 <bowlofeggs> but eventually i'd like to try bodhi in it, once others have gotten the infra feet wet :)
18:18:19 * threebean nods
18:18:31 <bowlofeggs> i can't see myself having time to focus on that myself in the near future
18:18:43 <nirik> side note: we still need to update our prod setup after freeze sometime
18:19:06 <threebean> mjia is working on putting out an update to waiverdb and greenwave this week.
18:19:14 <nirik> which might also involve just reinstalling it, because we may need to shuffle things around for some new drives.
18:19:19 <threebean> getting a feel for what sticking points there are in updating a service.
18:19:26 <relrod> yeah. I can do that. We'll need to announce an outage, but if it goes as smoothly as the stg one did, it shouldn't be very hard at all
18:19:45 <relrod> (of course now I probably jinxed it :P)
18:20:49 <nirik> threebean: sounds good.
18:20:55 <relrod> bowlofeggs: Well, you gave me a good segue for the Bodhi stuff (I'll go back up to Ticket cleanup after)
18:20:58 <relrod> #topic Can Bodhi move to be hosted on Fedora instead of Fedora and EL 7?
18:20:59 <relrod> #info The version of Pungi in EPEL 7 is quite old, and is causing some difficulties on https://github.com/fedora-infra/bodhi/pull/1697
18:21:06 <relrod> bowlofeggs: all yours
18:21:50 <bowlofeggs> heyo!
18:22:05 <bowlofeggs> ok, so there is a pull request to add pungi mashing to bodhi that's open right now
18:22:09 <bowlofeggs> linked in that #info above
18:22:33 <bowlofeggs> unfortunately, it is using internal pungi APIs that are only available on newer versions of pungi
18:22:45 <bowlofeggs> and the version of pungi available on EL 7 doesn't have those APIs
18:22:47 <nirik> well, bodhi-backend01 is already fedora. :)
18:22:51 <bowlofeggs> indeed!
18:23:02 <bowlofeggs> but, bodhi currently fully supports EL 7 and Fedora
18:23:12 <bowlofeggs> (keep in mind that there are other orgs that use bodhi too)
18:23:30 <bowlofeggs> and bodhi's tests currently all pass and have 90% coverage on EL 7 and fedora
18:23:33 <puiterwijk> bowlofeggs: so, how much promises do we have that the APIs they're using are going to continue to exist? If it's internal, that would be not promised
18:23:46 <bowlofeggs> puiterwijk: i don't think we have promises
18:23:51 <nirik> is there any chance that pungi is ok on epel7, they just didn't build it yet?
18:23:58 <threebean> nirik: I believe that's the case.
18:24:01 <bowlofeggs> puiterwijk: but, i think mcurlej is working to get an API added to pungi that bodhi could use
18:24:03 <threebean> nirik: "in theory"
18:24:42 <bowlofeggs> the tests kinda bother me, because bodhi's CI currently measures test coverage, but only runs on EL 7 today
18:24:57 <puiterwijk> bowlofeggs: and I'm going to assume that we're waiting with going to prod until we get stuff to use non-internal APIs?
18:25:06 <bowlofeggs> i kinda just feel conflicted about various options i can see, and wanted feedback from others
18:25:22 <puiterwijk> Anyway. Given that there's already Bodhi boxes on Fedora, I guess that that part is "fine"
18:25:42 <bowlofeggs> puiterwijk: i have a feeling that that current plan won't allow time for that to happen
18:25:44 <nirik> I have no objection to just switching all bodhi hosts to fedora if desired... but imho if we can keep supporting epel7 without great burden we should try to
18:25:58 <bowlofeggs> yeah i feel conflicted about switching it all to fedora
18:25:58 <puiterwijk> Agreed to nirik.
18:26:11 <bowlofeggs> on one hand, it works today and dropping support feels "bad"
18:26:17 <bowlofeggs> especially since i don't know how others deploy it
18:26:26 <nirik> does pungi even have a published api?
18:26:30 <bowlofeggs> but on the other hand, it does make this difficult
18:26:32 <bowlofeggs> nirik: it does not
18:26:35 <puiterwijk> nirik: nope
18:26:48 <bowlofeggs> nirik: but i think martin is trying to add one too
18:26:52 <puiterwijk> Which means they make no promises as to whether the APIs bodhi is going to call exist in 2 weeks
18:27:02 <bowlofeggs> threebean: is that still the plan? for martin to get this into pungi as a supported API?
18:27:11 <bowlofeggs> puiterwijk: correct
18:27:16 <threebean> he's working on it, yup.
18:27:18 <nirik> sure.
18:27:50 <bowlofeggs> so, for infra, i guess we could carry our own pungi package that's newer than EL 7 has? but then we have to stay on top of updates...
18:28:06 <bowlofeggs> and that might not be great for third party bodhi users, depending on how they deploy (unknown)
18:28:11 <nirik> I think first step is to ask lseldar if he can build for epel7
18:28:17 * threebean nods
18:28:21 <bowlofeggs> yeah that's an option too
18:28:30 <bowlofeggs> i guess that would be nicest
18:28:53 <threebean> if I had to bet, I'd say the hurdle is going to be python-kobo.  ;)
18:29:01 <nirik> hum...
18:29:07 <nirik> epel7 currently has pungi3
18:29:16 <nirik> ie, ancient
18:30:07 <bowlofeggs> anotehr crazy idea i had was trying to get some CI setup that can merge two coverage reports into an "uber coverage" report
18:30:17 <bowlofeggs> and then have a container to run the tests in F25+ too
18:30:31 <bowlofeggs> and then make sure we hit the new masher lines in at least one of the test runs
18:30:48 <bowlofeggs> and then skip tests related to pungi when running on EL 7
18:30:53 <bowlofeggs> but that seems pretty complicated
18:31:08 <bowlofeggs> and also still drops support for EL 7 for mashing, which again i don't know if others use today
18:31:22 <threebean> well, only for rpm mashing atm.
18:31:28 <smooge> meh just drop it
18:31:58 <nirik> oh, side note: bodhi-backend01* are f25, should sometime take them to f26.
18:32:06 <bowlofeggs> +1
18:32:34 <bowlofeggs> so i'd love to hear more ideas if anyone has them, but it sounds like asking lsedlar about getting newer pungi in EL 7 might be a good place to start?
18:32:52 <bowlofeggs> or, do we think just moving all of bodhi to F26 is better?
18:32:58 <bowlofeggs> or something else?
18:33:34 <nirik> perhaps dgilmore knows: is epel7 supposed to stay on pungi3 for some reason, or can it go to 4?
18:33:35 <puiterwijk> bowlofeggs: I don't think a new pungi would be in policy. Since the pungi3 -> pungi4 upgrade is probably pretty huge, so would need an EPEL exception
18:34:08 <nirik> puiterwijk: sure, but pungi3 isn't maintained anymore and I bet few people use it... so it could be done
18:34:12 <puiterwijk> (I'm assuming that they've changed input-output)
18:34:43 <nirik> they are... completely different.
18:34:58 <bowlofeggs> there is a policy that can be used to make backwards incompatible upgrades in EPEL that could be investigated, but i did it once and it does take some time (reasonably so)
18:35:19 <puiterwijk> Right. It just takes a while
18:35:35 <dgilmore> nirik: we could move it to 4
18:35:42 <bowlofeggs> so one problem is that we want to deploy this on the monday after the beta (so, like, 10-11 days?)
18:36:05 <dgilmore> nirik: we had some patches from scientific linux in the 3.x branch
18:36:15 <bowlofeggs> well, i should say that the modularity folks want to deploy it then. i would love to have more time to do things more cleanly than this.
18:36:18 <dgilmore> but at this point I think moving it makes sense
18:36:57 <nirik> well, if it's just a tide over we could have a pungi4 in infra tags
18:37:03 <bowlofeggs> moving it sounds good, but i doubt we can do that via the exception policy inside of 10 days
18:37:07 <dgilmore> I had attempted a pungi4 build for epel
18:37:08 <nirik> (until the epel one is out)
18:37:09 <dgilmore> https://koji.fedoraproject.org/koji/buildinfo?buildID=859976
18:37:23 <dgilmore> there was something missing
18:37:38 <bowlofeggs> nirik: true
18:37:44 <nirik> possibly the python-kobo threebean mentioned
18:38:39 <dgilmore> nirik: bowlofeggs: we will have to run pungi on fedora
18:38:51 <dgilmore> in order to get rich deps etc
18:38:53 <nirik> sure.
18:39:01 <bowlofeggs> yeah we do plan to keep doing that
18:39:01 <nirik> so perhaps we should just let el7 go
18:39:26 <dgilmore> nirik: as far as pushing updates in fedora/epel, yeah
18:39:28 <bowlofeggs> to be honest, the webob stuff in EL 7 does also cause me some pain from time to time
18:39:56 <bowlofeggs> i've always been able to work around those pains, but it has slowed me down before on more than one occasion
18:40:16 <bowlofeggs> so moving bodhi to pure fedora might make those pains lessen for me (but cause other pains, like more frequent upgrades)
18:40:17 <dgilmore> there are reasons why we can not fully support RHEL for our use cases
18:40:57 <nirik> well, I guess the question is: are any of the other bodhi users using on el7 only?
18:41:16 <bowlofeggs> so i personally think it would make my life a little nicer if bodhi was pure fedora, but i also know that would be more work for systems people so i want to make sure that wouldn't bother anyone
18:41:42 <bowlofeggs> nirik: yeah i also have that question and i really don't know the answer
18:42:25 <nirik> on infra end I don't mind switching them all to fedora. The web frontends seem kinda overkill to be fedora, but it should be fine.
18:42:27 <puiterwijk> bowlofeggs: that side of things is less of a problem, I'd say, given that we have multiple Fedora machines that we need to rebuild regularly anyway
18:42:39 <bowlofeggs> ah ok
18:42:51 <smooge> we should just make everything rawhide
18:43:02 <nirik> rollin rollin rollin...
18:43:12 <bowlofeggs> well, it would make my life a little easier and i think it could help with the problem at hand too
18:43:19 <bowlofeggs> any opposed?
18:43:28 <puiterwijk> Not opposed to making them Fedora, no
18:44:40 <bowlofeggs> ok, well thanks :)
18:44:51 <bowlofeggs> i think that satisfies what i wanted to talk about
18:45:03 <relrod> okay
18:45:19 <relrod> So I'll move back up to:
18:45:21 <relrod> #topic Ticket cleanup
18:46:15 <relrod> I'm not really sure what this is supposed to be/who added it, but -- is anyone working on tickets that they need to discuss or need help with?
18:47:10 <nirik> well, I was going to pick some old ones to discuss and move forward...
18:47:14 <nirik> but then I didn't. ;)
18:47:22 <nirik> I can look real quick
18:48:09 <nirik> .title https://pagure.io/fedora-infrastructure/issue/2079
18:48:09 <zodbot> nirik: Issue #2079: querying the list of all the aliases that a given FAS id is on - fedora-infrastructure - Pagure
18:48:23 <nirik> So, this has been around about 7 years.
18:48:36 <nirik> can we think of any way to actually implement it?
18:49:11 <nirik> I guess a grep of /etc/aliases /etc/postfix/* on bastion01 might...
18:49:13 <puiterwijk> nirik: well, "so that I can request to be removed from ones that I don't need to be on." - people can do a git grep through the ansible repo
18:50:10 <nirik> right. So, perhaps we close and add a howto clone ansible repo, git grep and look for aliases specifically?
18:51:28 <nirik> any objections? anyone want to do that? (I can if no one wants to)
18:52:09 <nirik> ok, shall we do another or call it good for today?
18:52:24 <puiterwijk> nirik: let's close all the tickets? :)
18:52:47 <nirik> sure, by fixing/addressing them. ;)
18:52:58 <nirik> here's another old one:
18:53:15 <nirik> .title https://pagure.io/fedora-infrastructure/issue/2888
18:53:15 <zodbot> nirik: Issue #2888: Implement HTTP Strict Transport Security - fedora-infrastructure - Pagure
18:53:39 <puiterwijk> That one's moved along step by step
18:53:44 <nirik> the only thing left here I think is to redirect http/https pkgs to src via iptables... I am not sure how well that will work
18:54:04 <puiterwijk> Let's not do it via iptables, but via apache.
18:54:08 <puiterwijk> That is simple enough to do.
18:54:34 <nirik> it can't be a redirect tho... that will fail
18:54:57 <nirik> well, http can be... but not https
18:55:26 <nirik> and upload.cgi still needs to work
18:56:07 <nirik> anyhow, we are low on time, can discuss more outside meeting
18:56:38 <relrod> yeah, only a few minutes left, so... I'm going to merge Apprentice office hours and regular open floor into one:
18:56:47 <relrod> #topic Open Floor and Apprentice Questions
18:57:08 <relrod> Anyone have anything they'd like to discuss, or apprentices have any quick questions they'd like to ask?
18:57:45 <benniej> All good here
18:57:52 <netcronin> Same. Working with the stg IP move.
18:58:04 <relrod> netcronin: *nod* - thanks for your help with all that!
18:58:07 <netcronin> Feel free to ping me on any issues you guys need help with that you think apprentices could tackle.
18:58:14 <netcronin> relrod: no problem. ):
18:58:16 <netcronin> :)
19:00:26 <relrod> okay, well, thanks all!
19:00:30 <relrod> #endmeeting