16:00:03 <phsmoura> #startmeeting Infrastructure (2023-04-27)
16:00:03 <zodbot> Meeting started Thu Apr 27 16:00:03 2023 UTC.
16:00:03 <zodbot> This meeting is logged and archived in a public location.
16:00:03 <zodbot> The chair is phsmoura. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:00:03 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:00:03 <zodbot> The meeting name has been set to 'infrastructure_(2023-04-27)'
16:00:04 <phsmoura> #meetingname infrastructure
16:00:04 <zodbot> The meeting name has been set to 'infrastructure'
16:00:04 <phsmoura> #chair nirik zlopez nb bodanel dtometzki jnsamyak
16:00:04 <zodbot> Current chairs: bodanel dtometzki jnsamyak nb nirik phsmoura zlopez
16:00:04 <phsmoura> #info Agenda is at: https://board.net/p/fedora-infra
16:00:04 <phsmoura> #info About our team: https://docs.fedoraproject.org/en-US/cpe/
16:00:04 <phsmoura> #info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
16:00:06 <phsmoura> #topic greetings!
16:00:10 <phsmoura> .hi
16:00:11 <zodbot> phsmoura: phsmoura 'Pedro Moura' <pmoura@redhat.com>
16:00:24 <nirik> morning everyone.
16:00:31 <leonkhan> morning
16:00:45 <darknao> .hi
16:00:46 <zodbot> darknao: darknao 'Francois Andrieu' <darknao@drkn.ninja>
16:02:17 <jednorozec> .hello humaton
16:02:18 <zodbot> jednorozec: humaton 'Tomáš Hrčka' <thrcka@redhat.com>
16:03:02 <mkonecny> .hello zlopez
16:03:03 <zodbot> mkonecny: zlopez 'Michal Konecny' <michal.konecny@pacse.eu>
16:03:35 <nb> .hi
16:03:36 * nirik welcome everyone
16:03:36 <zodbot> nb: nb 'Nick Bebout' <nick@bebout.net>
16:04:46 <phsmoura> #topic New folks introductions
16:04:46 <phsmoura> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
16:04:46 <phsmoura> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
16:05:07 <phsmoura> are there new people around? feel free to introduce yourselves
16:06:22 <phsmoura> #topic Next chair
16:06:22 <phsmoura> #info magic eight ball says:
16:06:22 <phsmoura> #info chair 2023-05-04 - lenkaseg
16:06:22 <phsmoura> #info chair 2023-05-11 - ??
16:06:22 <phsmoura> #info chair 2023-05-18 - ??
16:06:22 <phsmoura> #info chair 2023-05-25 - ??
16:06:53 <phsmoura> anyone willing to be chair in the available dates?
16:06:57 <lenkaseg> .hi
16:06:58 <zodbot> lenkaseg: lenkaseg 'Lenka Segura' <lenka@sepu.cz>
16:07:17 <nirik> I can take 5-11 if noone else wants it...
16:07:58 <phsmoura> ok, anyone wants 05-18 or 05-25?
16:08:11 <mkonecny> I can take the week after that
16:08:32 <phsmoura> mkonecny: 05-18?
16:08:38 <mkonecny> Yes
16:09:09 <phsmoura> ok, 05-25 anyone?
16:09:29 <erolkskn> .hello erolkeskin
16:09:30 <zodbot> erolkskn: erolkeskin 'Erol Keskin' <erolkeskin.dev@gmail.com>
16:10:04 <phsmoura> ok, this is what we have for the following weeks
16:10:05 <phsmoura> #info magic eight ball says:
16:10:06 <phsmoura> #info chair 2023-05-04 - lenkaseg
16:10:06 <phsmoura> #info chair 2023-05-11 - nirik
16:10:06 <phsmoura> #info chair 2023-05-18 - mkonecny
16:10:06 <phsmoura> #info chair 2023-05-25 - ??
16:11:11 <phsmoura> let me reload the board, its throwing me an error
16:11:39 <phsmoura> #topic announcements and information
16:11:39 <phsmoura> #info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1030 Europe/paris in #centos-meeting
16:11:39 <phsmoura> #info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in #fedora-meeting-3
16:11:39 <phsmoura> #info Fedora Linux 38 release is here! Here's how you can do a quick update: https://docs.fedoraproject.org/en-US/quick-docs/dnf-system-upgrade/
16:11:41 <phsmoura> #info Fedora 38 Freeze is over
16:11:55 <phsmoura> any other announcement?
16:12:02 <jednorozec> so
16:12:12 <jednorozec> we have moved the time of EU CPE standup
16:12:16 <jednorozec> -1h
16:12:29 <nirik> oh yeah, adjust that on the board... ;)
16:13:30 <mkonecny> jednorozec: We changed it to UTC as well
16:14:02 <mkonecny> #info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0730 UTC in #centos-meeting
16:14:30 <mkonecny> I updated it in agenda as well
16:15:45 <phsmoura> ok, moving on
16:15:49 <phsmoura> #topic Oncall
16:15:49 <phsmoura> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
16:15:49 <phsmoura> #info https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/
16:16:23 <phsmoura> #info jednorozec is on call from 2023-04-21 to 2023-04-27
16:16:24 <phsmoura> #info zlopez is on call from 2023-04-28 to 2023-05-04
16:16:24 <phsmoura> #info nirik is on call from 2023-05-05 to 2023-05-10
16:17:25 <phsmoura> #info ?? is on call from 2023-05-12 to 2023-05-18
16:17:25 <phsmoura> #info ?? is on call from 2023-05-19 to 2023-05-25
16:17:44 <phsmoura> anyone wants to take on call in the available weeks?
16:17:55 <jednorozec> sign me up for 19-25
16:18:04 <mkonecny> .oncalltakeeu
16:18:06 <zodbot> mkonecny: Kneel before zod!
16:18:10 <darknao> and me for 12-18
16:18:47 <phsmoura> ok, this is how it is now
16:18:48 <phsmoura> #info jednorozec is on call from 2023-04-21 to 2023-04-27
16:18:48 <phsmoura> #info zlopez is on call from 2023-04-28 to 2023-05-04
16:18:48 <phsmoura> #info nirik is on call from 2023-05-05 to 2023-05-10
16:18:48 <phsmoura> #info darknao is on call from 2023-05-12 to 2023-05-18
16:18:49 <phsmoura> #info jednorozecis on call from 2023-05-19 to 2023-05-25
16:18:57 <nirik> hurray.
16:19:00 <phsmoura> #info Summary of last week: (from current oncall )
16:19:03 <jednorozec> it was a quiet week i had no pings
16:19:18 <phsmoura> ok, next topic
16:19:30 <phsmoura> #topic Monitoring discussion [nirik]
16:19:31 <phsmoura> #info https://nagios.fedoraproject.org/nagios
16:19:31 <phsmoura> #info Go over existing out items and fix
16:20:09 <nirik> Not much new here... about the same as it has been. We are still seeing some FMN alerts from time to time... might have to look into it again if it keeps happening.
16:20:27 <nirik> Oh, and there's a anoying registry alert that we should fix.
16:20:30 <mkonecny> What about the builder alerts?
16:20:58 <nirik> From time to time, builders get wedged and need power cycling. It seems to be happening more lately for some reason.
16:21:41 <nirik> the registry thing is pretty complex.. I could write up whats going on if some folks want to look at it and see what I missed fixing it.
16:22:30 <nirik> The short version is:
16:23:22 <mkonecny> Thanks for the explanation about the builders, I recently saw plenty of alerts and wasn't sure what to do with them
16:24:11 <nirik> we have a golang thing called 'reg' that runs on sundries01. It generates a html index file based on the containers in the registry. It writes this locally. Proxies then sync this content out. This is the page you get when you go to registry.fedoraproject.org in a browser. It's only that. Sometimes, reg seems to generate a 0 length file and the proxies sync it... then it's broken until the next time they sync. So, we could put in some way to see that
16:24:11 <nirik> it's 0 length and not sync. or fix reg. or something.
16:24:54 <nirik> we have been having problems with the power9 boxes... since f27 or so they have sometimes just stopped responding and all the buildvm-ppc64le builders on that affected box have problems. ;(
16:25:14 <nirik> I've tried upgrading bvmhost-p09-03 to f38 and newer kernel to see if it helps.
16:25:19 <darknao> is there a ticket created about the reg issue?
16:25:49 <darknao> and aren't we planning to move to quay at some point? is there any help needed on that front?
16:25:57 <nirik> nope, but I can make one.
16:26:27 <nirik> yes, but we need to investigate that quay can do all the things we need. (arches, flatpaks, etc)
16:27:08 <darknao> maybe we can just try to mirror our registry to quay, and see what's failing?
16:27:11 <nirik> sundries01 is currently rhel7... so it's pretty ancient.
16:28:18 <nirik> perhaps. We do already have stuff there, so we wouldn't want to break that... but if we could push the copy somewhere to test that would be good info
16:28:57 <nirik> we could also replace reg or... run it in a openshift container so we can just run a newer version.
16:29:23 <erolkskn> is this 'reg' thing written in golang ?
16:29:30 <erolkskn> if so i could help ?
16:30:11 <nirik> ha... we are running the latest version.
16:30:12 <nirik> it was last updated in 2019
16:30:20 <nirik> https://github.com/genuinetools/reg
16:30:31 <nirik> so, perhaps we could find some replacement?
16:31:41 <nirik> Or you know, I am not sure what people use the web page for... perhaps we could replace it with a static thing?
16:32:51 <nirik> for one thing it has all the flatpaks listed... but you can't 'docker pull' a flatpak and just use it as far as I know?
16:33:33 <nirik> so, how about a email thread (or discussion thread?) on this... and we can figure out what we want to do?
16:33:49 <nirik> we can rope in the flatpak folks.
16:34:13 <darknao> we have a request here on the website repo: https://gitlab.com/fedora/websites-apps/fedora-websites/fedora-websites-3.0/-/issues/81 so maybe we can create a page on the new website with the listing of the images that most users are looking for
16:34:36 <darknao> not sure if that could be a good replacement
16:35:59 <nirik> might be. I am not fully sure what people expect going to https://registry.fedoraproject.org/ ;)
16:36:00 <nirik> it also lists the ostree desktops... and a docker pull of those I don't think would get you very far. ;)
16:36:31 <nirik> I like the idea of just redirecting it to a container page on websites... showing base minimal and toolbx
16:38:51 * nirik can start a discussion of this. Don't want to hyjack the entire meeting. ;)
16:38:57 <phsmoura> today we have a learning topic should we skip to that or do backlog refinement?
16:39:57 <mkonecny> Let's do learning topic when we have it :-)
16:40:34 <phsmoura> #topic Upcoming learning topics
16:40:34 <phsmoura> #topic Sysadmin War Story Discussion [all_of_us] on 2023-04-27
16:41:54 <nirik> ah yeah.
16:42:30 <nirik> who has stories to tell? :) chime in...
16:43:06 <erolkskn> nirik: (reg) it seems like it just fetches the container image information from specified registry and builds up an index page for them (to search, show container details and so on )
16:43:32 <nirik> yes, thats it. ;)
16:44:05 <nirik> I have so many stories, I am not sure where to start even.
16:44:27 <mkonecny> I have one I can remember right now
16:44:42 <mkonecny> nirik: Do you want to start?
16:45:01 <nirik> I have an amusing one from my college days so very long long ago...
16:46:16 <nirik> So, in college I helped run a recreational computing server. Basically a gaming server. We had a donated microvaxII running ultrix and some of us set it up and managed accounts for people to login and run various games. We got people from all over via word of mouth, there was no advertising.
16:46:46 <nirik> So, the university required us to keep records of accounts and ask people for ID (they would mail us... snail mail!) a form with their id, etc.
16:47:07 <nirik> Things were going all great. Some drama and a few issues, but overall everyone was having fun.
16:48:34 <nirik> Then... finals week. I was just about to go to a final test for a class and... someone came over and said "Hey, the FBI is looking for you" :)
16:48:34 <nirik> and indeed, turns out someone was using our little games machine to attack a gov site. :(
16:48:34 <nirik> so, they took out storage away and we had to reinstall everything from scratch
16:48:51 <nirik> So, always remember security is important. :)
16:49:32 <nirik> These days I am sure they wouldn't have sent someone physically out to take our drive... but they sure did back then. ;)
16:49:50 <nirik> EOF
16:50:01 <mkonecny> Good story :-) Reminds me of my last day in my first work
16:50:39 <mkonecny> I arrived in the morning and found out that the offices are sealed and the police is searching them
16:51:39 <mkonecny> So I didn't get to the office and we worked from nearby college
16:52:26 <mkonecny> We didn't get much done, because there wasn't a way to connect to our servers outside of the offices
16:52:36 <mkonecny> But this isn't the point of the story
16:53:09 <mkonecny> The point is that my collegues had goodbye gift for me in the office
16:53:57 <mkonecny> So they went and two of them get inside the offices with armed police man as a company
16:54:38 <mkonecny> They arrived to the lockers, got the gift for me and were escorted back to the hall
16:54:54 <mkonecny> I must say that was a crazy last day at work :-D
16:55:20 <mkonecny> So this is not really sysadmin story, but the nirik story reminded me of that one
16:55:32 <nirik> If only they could have waited you would never have to have had to deal with it. ;)
16:55:44 <erolkskn> But why was police searching the building ?
16:56:11 <mkonecny> It was some kind of pre-election rivalry, owner of the company had senator brother
16:56:15 <nirik> Or did you ever find out?
16:56:23 <erolkskn> Oh :D
16:56:31 <erolkskn> Good stories
16:56:34 <nirik> (we never got our hard drive back from the FBI. ;)
16:57:21 <leonkhan> :)
16:57:39 <mkonecny> I still see the company name mentioned when there is some kind of sh*t happening in the goverment IT :-D
16:58:01 <leonkhan> is it true that the visitors of the FBI websites are tracked via IP address ?
16:58:35 <mkonecny> I would say that every website is tracking your IP address nowadays
16:58:36 <nirik> no idea. I mean, probibly their webserver logs the ips of requests like anywhere?
16:58:52 * nirik sees we are running low on time.
16:59:38 <phsmoura> yep, so should we close it?
17:00:35 <phsmoura> seems theres nothing to add anymore
17:00:37 <mkonecny> Yes, we don't want to block the channel :-)
17:00:54 <phsmoura> thank you all for joining :)
17:00:57 <phsmoura> #endmeeting