16:01:49 <petebuffon> #startmeeting Infrastructure (2021-10-28)
16:01:50 <zodbot> Meeting started Thu Oct 28 16:01:49 2021 UTC.
16:01:50 <zodbot> This meeting is logged and archived in a public location.
16:01:50 <zodbot> The chair is petebuffon. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:01:50 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:01:50 <zodbot> The meeting name has been set to 'infrastructure_(2021-10-28)'
16:01:54 <petebuffon> #meetingname infrastructure
16:01:54 <zodbot> The meeting name has been set to 'infrastructure'
16:01:59 <eddiejennings> .hello
16:01:59 <zodbot> eddiejennings: (hello <an alias, 1 argument>) -- Alias for "hellomynameis $1".
16:02:00 <petebuffon> #chair nirik siddharthvipul mobrien zlopez pingou bodanel dtometzki jnsamyak computerkid
16:02:00 <zodbot> Current chairs: bodanel computerkid dtometzki jnsamyak mobrien nirik petebuffon pingou siddharthvipul zlopez
16:02:01 <nirik> morning
16:02:05 <petebuffon> #info Agenda is at: https://board.net/p/fedora-infra
16:02:09 <petebuffon> #info About our team: https://docs.fedoraproject.org/en-US/cpe/
16:02:09 <Leo[m]1234> .hi leo
16:02:10 <zodbot> Leo[m]1234: Sorry, but user 'Leo [m] 1234' does not exist
16:02:12 <petebuffon> #topic greetings!
16:02:15 <darknao> .hi
16:02:16 <zodbot> darknao: darknao 'Francois Andrieu' <darknao@drkn.ninja>
16:02:17 <LenkaSegura[m]> .hello lenkaseg
16:02:19 <zodbot> LenkaSegura[m]: lenkaseg 'Lenka Segura' <lenka@sepu.cz>
16:02:20 <petebuffon> .hello petebuffon
16:02:22 <zodbot> petebuffon: petebuffon 'Peter Buffon' <pabuffon@gmail.com>
16:02:23 <petebuffon> hey everyone :)
16:02:23 <eddiejennings> .hi
16:02:25 <zodbot> eddiejennings: eddiejennings 'Eddie Jennings' <eddie@eddiejennings.net>
16:02:28 <Leo[m]1234> .hello leo
16:02:29 <zodbot> Leo[m]1234: leo 'Leo Puvilland' <lpuvilla0001@mymail.lausd.net>
16:02:30 <AidenLangley[m]> /wave
16:03:14 <petebuffon> #topic New folks introductions
16:03:15 <petebuffon> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
16:03:15 <petebuffon> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
16:03:22 <austinpowered> .hi
16:03:22 <zodbot> austinpowered: austinpowered 'T.C. Williams' <fedoraproject@wootenwilliams.com>
16:03:24 <petebuffon> any newcomers today?
16:03:40 <AidenLangley[m]> I rambled enough in my email ^^
16:03:47 <AidenLangley[m]> but yes, me, hey everybody
16:03:58 <Leo[m]1234> hello :D welcome
16:04:18 <petebuffon> welcome! if you want please feel free to introduce yourself.
16:04:35 <darknao> welcome AidenLangley[m]
16:04:57 <nirik> welcome AidenLangley[m]
16:05:03 <AidenLangley[m]> tyty, yeah briefly, I'm nedia or aiden, sometimes langs 🙂 it's 5am, I might sleep after this haha but I'm a night owl
16:05:22 <pmoura> welcome AidenLangley[m]
16:05:25 <eddiejennings> welcome!
16:05:27 <AidenLangley[m]> I'm just getting into fedora & open source and trying to see where I can be useful
16:05:32 <leothecat> alr am on irc now
16:05:39 <leothecat> but i'm Leo on matrix :)
16:05:41 <siddharthvipul> \o
16:05:52 <siddharthvipul> .hello siddharthvipul1
16:05:53 <zodbot> siddharthvipul: siddharthvipul1 'Vipul Siddharth' <siddharthvipul1@gmail.com>
16:06:08 <petebuffon> wow 5am is early
16:07:01 <petebuffon> well welcome welcome, if you have any questions feel free to ask on fedora-admin, we'd be happy to answer!
16:07:19 <AidenLangley[m]> I grew up in London but now in NZ, I think a part of me is still on the other side of the planet so I am always up far too late
16:07:42 <AidenLangley[m]> petebuffon: ty will do
16:07:50 <leothecat> hello! :D
16:08:01 <petebuffon> I spent some time in NZ and really liked it
16:08:22 * nirik has heard good things, but never been there.
16:08:23 <AidenLangley[m]> it's lovely
16:08:57 <petebuffon> alright let's move on to the next chair
16:09:02 <petebuffon> ### Determine who the next chair is
16:09:02 <petebuffon> #topic Next chair
16:09:03 <petebuffon> #info magic eight ball says:
16:09:03 <petebuffon> #info chair 2021-11-04 -  lenkaseg
16:09:03 <petebuffon> ##info chair 2021-11-11 - ???
16:09:03 <petebuffon> ##info chair 2021-11-18 - ???
16:09:15 <petebuffon> any takers for the 11-11 or 11-18?
16:09:40 <eddiejennings> I can't do 11/11 because I might not make the meeting, but I can do 11/18
16:10:06 <petebuffon> sounds good eddiejennings
16:10:19 <petebuffon> ##info chair 2021-11-18 - eddiejennings
16:10:21 * nirik can do 11-11 if no one else wants it.
16:11:09 <petebuffon> and it's yours!
16:11:15 <petebuffon> ##info chair 2021-11-11 - nirik
16:11:34 <petebuffon> perfect let's move on
16:11:36 <petebuffon> #topic announcements and information
16:11:36 <petebuffon> #info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1030 Europe/paris in #centos-meeting
16:11:36 <petebuffon> #info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in #fedora-meeting-3
16:11:36 <petebuffon> #info If your team wants support from the Fedora Program Management Team, file an isssue: https://pagure.io/fedora-pgm/pgm_team/issues?template=support_request
16:11:38 <petebuffon> #info Fedora 35 Beta was released on 2021-09-28
16:11:40 <petebuffon> #info Fedora 35 Final freeze is in effect
16:11:49 <leothecat> yep
16:11:52 <petebuffon> any other news topics for today?
16:12:59 <nirik> #info f35 go/no-go in about 45min (after this meeting)
16:13:58 <petebuffon> perfect
16:14:19 <leothecat> cool
16:14:43 <petebuffon> alright oncall time then
16:14:44 <petebuffon> #topic Oncall
16:14:45 <petebuffon> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
16:14:45 <petebuffon> #info https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/
16:14:45 <petebuffon> #info nirik on call from 2021-10-21 to 2021-10-28
16:14:46 <petebuffon> ## .oncalltakeeu .oncalltakeus
16:14:48 <petebuffon> #info darknao on call from 2021-10-28 to 2021-11-04
16:14:50 <petebuffon> #info eddiejennings on call from 2021-11-04 to 2021-11-11
16:14:52 <petebuffon> #info ??? on call from 2021-11-11 to 2021-11-18
16:15:07 <petebuffon> any takers for on call starting the week of 11-11?
16:15:33 <darknao> .oncalltakeeu
16:15:35 <zodbot> darknao: Kneel before zod!
16:15:41 <ahmedalmeleh> what is that call for?
16:16:10 <petebuffon> the person who is on call for the week accepts pings for people who need help.
16:16:19 <eddiejennings> If no one else wants, I can do a second week (11/11). I generally like ending my on-call when the week I chair. :D
16:16:37 <leothecat> wait lemme look at my calendar
16:16:50 <petebuffon> they then triage the help needed and help out if they can, make a new ticket, or if necessary alert other team members in an emergency
16:17:03 <leothecat> i can probably do it
16:17:04 <ahmedalmeleh> got it
16:17:08 <petebuffon> check out https://fedoraproject.org/wiki/Infrastructure/Oncall
16:17:42 <petebuffon> makes sense eddiejennings
16:17:50 <petebuffon> alright so i'll give it to leothecat then
16:17:51 <LenkaSegura[m]> I can take oncall too some week.
16:18:31 <petebuffon> #info leothecat on call from 2021-11-11 to 2021-11-18
16:18:34 <leothecat> yep
16:18:39 <eddiejennings> leothecat: Would you be willing to do 11/4's week, and I take 11/11?
16:19:02 <leothecat> uh probably?
16:19:23 <petebuffon> okay
16:19:38 <petebuffon> #info leothecat on call from 2021-11-04 to 2021-11-11
16:19:43 <eddiejennings> Thank you :D
16:19:54 <ahmedalmeleh> I want to join infrastructure's team
16:19:54 <petebuffon> #info eddiejennings on call from 2021-11-11 to 2021-11-18
16:20:20 <petebuffon> LenkaSegura want to take the week after, 11-25?
16:21:19 <eddiejennings> I hear you get time and a half when on-call on Thanksgiving ;)
16:21:20 <LenkaSegura[m]> ok!
16:21:39 <petebuffon> hah :) kk
16:21:57 <LenkaSegura[m]> in Spain we give hanks all year :)
16:22:07 <LenkaSegura[m]> s/hanks/thanks
16:22:14 <petebuffon> #info LenkaSegura on call from 2021-11-25 to 2021-12-02
16:22:28 <petebuffon> moving on
16:22:29 <petebuffon> #info Summary of last week: (from current oncall )
16:22:29 <petebuffon> #topic Monitoring discussion [nirik]
16:22:29 <petebuffon> #info https://nagios.fedoraproject.org/nagios
16:22:29 <petebuffon> #info Go over existing out items and fix
16:23:01 <nirik> There were no oncall calls that I saw. I think mostly people are trying to get f35 out the door... :)
16:23:07 <nirik> for monitoring...
16:23:10 <leothecat> a service is flapping :P
16:23:38 <nirik> yeah, thats a bogus check we need to remove...
16:24:04 <nirik> otherwise it's pretty much the same as last week... and we are freeze, so we haven't really fixed anything
16:24:12 <nirik> we can move on unless there's questions...
16:24:44 <petebuffon> release purgatory
16:25:14 <nirik> we have a lot of things piled up for after freeze. Gonna be fun.
16:25:23 <leothecat> yep
16:25:37 <petebuffon> nice
16:25:51 <petebuffon> well I believe you're up again nirik, if you're ready for the learning topic
16:25:58 <nirik> ah yeah.
16:26:05 <petebuffon> #topic Learning topic
16:26:10 <petebuffon> #info 2021-10-28 - Introduction to AWS and Fedora Infrastructure [nirik]
16:26:33 <nirik> so, lots to cover here, so feel free to ask questions if I don't cover something you would like me to...
16:26:37 <ahmedalmeleh> are we using AWS?
16:26:48 <nirik> we are for some things. :)
16:26:51 <copperi[m]1> yes
16:26:57 <leothecat> oo
16:27:05 <leothecat> im interested in this
16:27:12 <AidenLangley[m]> me too
16:27:17 <ahmedalmeleh> me three
16:27:37 <nirik> So, amazon has very gracefully provided us with a community account. This lets us do things for our project and amazon takes care of the bill. Kudos to amazon for that.
16:27:54 <ahmedalmeleh> Good on them
16:28:04 <AidenLangley[m]> besoz can spare a penny or two
16:28:08 <nirik> We only have the one account thats setup this way and we need to share it for a lot of things that we don't want to interfere with each other.
16:28:40 <nirik> We have setup authentication on this account to use our auth system (via SAML2) and groups.
16:28:46 <ahmedalmeleh> Okay
16:29:12 <nirik> The aws-* groups in our account system map to permissions in aws. So, someone in that group logs in via our auth and gets logged in as that group.
16:29:34 <leothecat> oh cool so they pay
16:29:39 <nirik> We then take advantage of IAM policies. We start with "deny everything" and add only those permissions that group needs to do what they need to do.
16:30:02 <nirik> There's a number of groups involved:
16:30:22 <nirik> * copr - copr uses this for builders and their frontend/backend boxes.
16:30:49 <nirik> * centos - centos uses this for instances and also cloudfront (which I will get to in a few)
16:31:14 <nirik> *fedora-ci uses this for a eks cluster that does ci tests on fedora packages
16:31:32 <nirik> * fedora-infra runs maintainer-test instances in it.
16:32:06 <nirik> * fedora infra runs some proxy servers in it (in regions where we don't have any donated servers, like APAC, AU, etc)
16:32:25 <nirik> * we also use it for some one off development/test instances when someone is testing something.
16:32:41 <leothecat> is it used for quick spinup and then destroy machines?
16:32:44 <leothecat> like to test one thing
16:32:45 <nirik> There may be more that I am not thinking of. ;)
16:33:04 <nirik> leothecat: nope, only for persistent stuff currently.
16:33:11 <leothecat> oh okay
16:33:31 <nirik> well, copr manages it's own instances... it spins up builders to do builds then destroys them when the build is done.
16:33:42 <nirik> it's pretty cool. :)
16:34:19 <nirik> Another big use we have with amazon is cloudfront. cloudfront is their caching proxy / cdn product.
16:34:43 <leothecat> phew am back i disconnected
16:34:46 <nirik> We have a cloudfront setup for registry.fedoraproject.org (our container registry)
16:34:58 <nirik> and ostree repos
16:35:17 <nirik> and we also have one for internal to aws fedora/epel use
16:35:17 <ahmedalmeleh> ok
16:35:26 <nirik> centos also is using cloudfront for mirrors and such.
16:35:36 <leothecat> okay
16:35:47 <leothecat> and all of this except copr is all done manually?
16:35:50 <leothecat> *for now*
16:36:11 <nirik> cloudfront works by getting requests and then fetching data from an origin server, then caching that locally. it has endpoints in basically everyplace amazon has regions.
16:37:03 <nirik> well, each group controls their things... copr, centos, fedora-ci all manage their resources however they want... I think for example fedora-ci uses an amazon deployment thing for eks
16:37:12 <leothecat> o okay
16:37:19 <petebuffon> Is there a specific reason for using cloudfront for registry.fedoraproject.org?
16:37:21 <nirik> but in fedora-infra, yes, we currently manually deploy things then manage them via ansible
16:37:32 <nirik> petebuffon: it gets a LOOOOOOOOT of traffic.
16:37:40 <petebuffon> got it
16:37:59 <nirik> basically podman on every fedora box hits it first and asks for whatever image...
16:38:10 <nirik> I think it's first in the search path
16:38:25 <copperi[m]1> nice
16:38:26 <petebuffon> right, I may have done that a few times...
16:39:16 <nirik> and for ostree things, we don't mirror them in our normal mirror network, so ostree using folks were seeing it pretty slow from other parts of the world...
16:39:59 <nirik> And finally we also have a app called fedimg that uploads our fedora images to aws...
16:40:31 <nirik> It basically uploads everything thats composed and passed a simple 'does it boot' test, and it copies that to every region.
16:40:53 <nirik> but it also does that for final composes, etc... they are all there.
16:41:59 <nirik> So if you ever want to fire off a fedora instance, https://alt.fedoraproject.org/cloud/ has buttons to do that. ;)
16:42:27 <leothecat> oh wow it just launches aws
16:42:30 <nirik> I think thats about all... any questions?
16:42:50 <AidenLangley[m]> fedimg hehe nice name... Can picture the logs reading 'fedimg: fed image to aws'
16:43:11 <copperi[m]1> nice info nirik
16:44:16 <nirik> ok, thanks everyone. Hopefully that made things as clear as mud. :)
16:44:18 <austinpowered> nirik++
16:44:24 <petebuffon> nirik++
16:44:25 <leothecat> cool :)
16:44:27 <AidenLangley[m]> is this a new set up?
16:44:30 <eddiejennings> nirik++
16:44:41 <eddiejennings> nirik keeps adding to my list of need-to-learn-and-become-familiar :P
16:44:41 <pmoura> nirik++
16:44:45 <leothecat> by the way the aws topic on the list was replied again
16:45:28 <nirik> AidenLangley[m]: nope, been around for years.
16:46:01 <copperi> nirik++
16:46:47 <nirik> petebuffon: back to you for open floor. ;)
16:46:51 <petebuffon> great! makes me want to dive into AWS
16:46:54 <petebuffon> #topic Open Floor
16:47:00 <petebuffon> got about 15 minutes left
16:48:23 <austinpowered> the learning topic for 2021-09-02 was ssh host keys signing and ansible interaction
16:48:52 <austinpowered> meetbot doesn't have any log files for that date
16:50:02 <AidenLangley[m]> so is somebody tasked w/ learning about the topic and giving the team a lesson?
16:50:07 <eddiejennings> I think meetbot and its logging has been broke for a while, no?  The logging itself isn't broken but the display on the web is, right?
16:51:14 <petebuffon> Maybe something to do with the zodbot changeover to python3?
16:51:20 <nirik> it should be working.
16:51:34 <eddiejennings> Ah.  I honestly haven't checked in a while.
16:51:46 <austinpowered> the dir at https://meetbot-raw.fedoraproject.org/teams/infrastructure/ skips from 08-19 to 09-09
16:51:49 <leothecat> so about the aws ansible integration
16:51:59 <leothecat> are we going to go ahead with it or wiat until freeze?
16:52:01 <nirik> or perhaps it was broken then...
16:52:34 <petebuffon> AidenLangley: the last portion of the meeting is either a learning topic or backlog refinement. It goes back and forth every week
16:52:48 <austinpowered> As I recall it was broken at the time. If the file available anywhere else?
16:53:00 <austinpowered> /If/Is/
16:53:07 <eddiejennings> Is https://pagure.io/fedora-docs/docs-fp-o now the official place to create / update infra documentation?
16:53:44 <nirik> austinpowered: I am looking
16:54:05 <austinpowered> nirik: thanks
16:55:40 <leothecat> so we wait until freeze for aws->ansible?
16:56:07 <nirik> I think we can work on it... PR's would surely be fine before freeze is over.
16:56:18 <nirik> eddiejennings: yes
16:56:33 <eddiejennings> excellent
16:56:42 <ahmedalmeleh> ok
16:57:30 <nirik> austinpowered: https://meetbot-raw.fedoraproject.org/fedora-meeting-3/2021/fedora-meeting-3.2021-09-02-16.00.html
16:57:40 <nirik> it looks like it didn't get the right name somehow
16:58:33 <leothecat> okay we can kind of copy from https://pagure.io/fedora-infra/arc/blob/main/f/ansible
16:58:55 <nirik> copy?
16:59:31 <leothecat> or well base it on
16:59:48 <leothecat> because they already use aws in ansible
17:00:16 <nirik> the aws provisioning? no need to copy the repo... it should just be adjustments to the tasks/cloud_setup_basic or whatever it is. ;)
17:00:22 <petebuffon> thanks everyone, if you want to keep the conversations going please move over to #fedora-admin
17:00:27 <petebuffon> #endmeeting