18:00:21 <pboy> #startmeeting fedora-server
18:00:21 <zodbot> Meeting started Wed Jan 18 18:00:21 2023 UTC.
18:00:21 <zodbot> This meeting is logged and archived in a public location.
18:00:21 <zodbot> The chair is pboy. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
18:00:21 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:00:21 <zodbot> The meeting name has been set to 'fedora-server'
18:00:29 <pboy> #topic Welcome / roll call
18:00:37 <pboy> Welcome to our Server WG IRC meeting today!
18:00:38 <mowest> .hello2
18:00:39 <zodbot> mowest: mowest 'Steve Daley' <mowest@vivaldi.net>
18:00:49 <pboy> Please, everybody who is lurking, say either .hello2 or .hello <fasname>
18:00:57 <pboy> I’ll post the agenda in a few minutes.
18:01:20 <copperi[m]> either .hello2 or .hello <fasname>
18:02:24 <cooltshirtguy> .hello2
18:02:28 <zodbot> cooltshirtguy: cooltshirtguy 'Jason Beard' <jas_beard@hotmail.com>
18:03:20 <pboy> Welcome everybody!
18:03:37 <pboy> I think, we should start now.
18:03:50 <pboy> #topic Agenda
18:04:11 <pboy> #info Follow up actions
18:04:26 <pboy> #info Shorter Shutdown Timer
18:04:40 <pboy> #info Using Ansible to install and configure Wildfly
18:04:52 <pboy> #info Server WG work program for upcomming year
18:05:03 <pboy> #info Open floor
18:05:11 * nirik is around for once.
18:05:12 <pboy> Anything to add?
18:05:28 <pboy> Welcome nirik, we may need your advice!
18:05:33 <mowest> Could I give an Fedora Server website update?
18:05:56 <nirik> I had thoughts on the shutdown thing (although perhaps not popular ones. ;)
18:06:05 <pboy> mowest I add that as new topic 3?
18:06:18 <eseyman> hello, folks. Sorry I'm late
18:06:25 <eseyman> .hello2 eseyman
18:06:26 <zodbot> eseyman: eseyman 'Emmanuel Seyman' <emmanuel@seyman.fr>
18:06:32 <pboy> eseyman we are used to.  :-)
18:07:02 <jwhimpel> .hello2
18:07:03 <zodbot> jwhimpel: jwhimpel 'John Himpel' <john@jlhimpel.net>
18:07:07 <pboy> OK, first topic
18:07:20 <pboy> #topic Follow up actions
18:07:34 <pboy> That's easy, there a none, as far as I know
18:07:43 <pboy> anything to add?
18:07:53 <mowest> No lets move on
18:08:07 <pboy> #topic Shorter Shutdown Timer
18:08:21 <pboy> I'll give a summary info at first.
18:08:47 <pboy> the decision: #agreed Change is approved with a timeout of 45 s and the caveat that editions must be able to override the change (+8,0,-1)
18:09:11 <pboy> Strategy is: aggressivly cut down the timeout threshold waiting for users to complain about lost data, corrupted file systems, inconsistent databases, etc,
18:09:20 <pboy> And then eventually enlarge the timeout threshold again.
18:09:28 <pboy> Question is: what do we want to do?
18:09:36 <pboy> 1. Take the 45 sec and wait for user complains (hoping they complain berfor switching to another repo)
18:09:39 <nirik> well, I am not sure thats how I would say it. ;)
18:09:48 <pboy> 2. Overwrite the value by either the old or a slightly lower
18:09:56 <pboy> 3. Tke the 45 sec and document how to adjust, hoping everybody reads the docs before a catastrophe
18:10:07 <mowest> Honestly, in a homelab environment which is my area of use for Fedora Server the current shutdown timer has been fine for my use case.
18:10:12 <pboy> nirik it's a quotation from zbyszek
18:10:25 <nirik> well, some things to consider:
18:10:43 <mowest> I'm not even sure what the current shutdown timer is at this point. Is it 60 seconds?
18:10:48 <nirik> all the server services I can think of that take a long time to shutdown cleanly... _already_ override the 2min timeout.
18:10:48 <pboy> nirik go on
18:11:12 <nirik> libvirtd, postgres, all of them are set to unlimited timeout already because 2min was too low for some cases.
18:11:47 <nirik> so really this affects things that take longer then 45s but less then 2min... I am not sure what would be in there thats not already overridden.
18:12:19 <pboy> nirik I'm not against a shorter threshold.
18:12:30 <jwhimpel> is the override programmatic or is it in the systemd .service file?
18:12:34 <pboy> But I wohl like to check all ourt core services first
18:12:44 <nirik> jwhimpel: in the service file.
18:12:56 <jwhimpel> nirik: Thanks
18:13:01 <pboy> jwhimpel there is a configuration option in /etc/systemd/system.conf
18:13:12 <eseyman> yes, the override is service-specific
18:13:18 <nirik> so, I would advocate that we stay with the 45s default until/unless we see a show stopper... since it's just easier to not diverge.
18:13:29 <pboy> no, this one is system wide default
18:13:38 <mowest> +1 @nirik
18:13:41 <nirik> pboy: no, each service can override it.
18:13:57 <nirik> ie, postgresql sets it to 0 (which is "no limit")
18:13:59 <pboy> nirik yes, can, but does it?
18:14:01 <nirik> as does libvirtd
18:14:09 <pboy> or did it just rely on the default?
18:14:32 <nirik> ie, look at postgresql:
18:14:34 <nirik> # No artificial start/stop timeout (rhbz#1525477, pgrpms#2786).
18:14:34 <nirik> TimeoutSec=0
18:14:52 <nirik> so, it will override the system default and never kill postgres.
18:14:57 <eseyman> on devel@, pboy suggested testing sample usecases. This seems a practical thing to do
18:15:16 <nirik> yes, I think a test day or the like would be a good idea.
18:15:23 <pboy> nirik yes postgres and libvirt do, what about samba, nfs, etc?
18:15:43 <jwhimpel> wildfly also
18:16:19 <nirik> well, my pont is that all these probibly already hit the 2min timeout and overrode it. :)
18:16:29 <nirik> I don't know specifically on those, can look.
18:17:36 <pboy> We have a new version of our technical specification. We should look up all services mentioned there
18:17:55 <nirik> sounds like a good idea, yes.
18:18:35 <pboy> The famous question: who will do it?
18:19:21 <nirik> I can try, but not sure how much time I have...
18:19:36 <nirik> could we ask for a test day and get testers interested in helping?
18:19:58 <eseyman> nirik: I will be glad to contribute to a test day
18:20:00 <pboy> #action nirik will look up the services in our technical specification and check for time out value
18:20:16 <pboy> hirik Thanks!
18:20:21 * nirik nods
18:20:23 <pboy> nirik Thanks!
18:20:44 <jwhimpel> Sorry, I have to leave early.  My wife just got home and is taking me to urgent care.  Thinking pneumonia.  Sorry!
18:21:03 <pboy> jwhimpel. Good luck!
18:21:15 <nirik> yikes. Hope everything is ok.
18:21:16 <pboy> Other question:
18:21:45 <pboy> Does server really benefit from a shorter default value?
18:22:05 <nirik> well, I think in 2 ways...
18:22:18 <eseyman> I think we do benefit
18:22:19 <nirik> one is just not diverging from the rest of fedora... so, less need for docs, etc
18:22:40 <nirik> but the other is that sometimes people do want a quick reboot (even though thats a dream with servers)
18:22:46 <eseyman> shorter shutdowns/reboots reduces downtime
18:22:56 <nirik> yeah...
18:23:21 <pboy> shorter downtime, as long as nothing goes wrong
18:23:37 <nirik> even though we have all sat watching "checking memory... 1k"
18:23:44 <mowest> I like quick reboots in homelab because you are often testing things out and messing things up, good to know you messed up fast instead of waiting for a long response.
18:23:46 <pboy> but when some thing breacks:
18:24:05 <mowest> It breaks quick :-)
18:24:27 <pboy> Yeah. :-). I like safe reboots in production
18:25:05 <eseyman> yes, some things will suffer corruption but a 2min timeout isn't that much better in that regard
18:25:14 <pboy> Interesting is the argumentation of Collin Wolters, that is the same as my concern.
18:25:33 <cooltshirtguy> i like the shorter shutdown time but I do understand that services might not react well to it
18:26:14 <pboy> It I get a shorter shutdown, but the a longer repair time ....
18:26:20 <pboy> nothing is gained.
18:26:46 <eseyman> let's remember that 2mins was just a conservative value determined because none existed beforehand
18:27:06 <eseyman> so I wouldn't call the current situation perfect either
18:27:17 <pboy> eseyman yes, bit at least we had no complains
18:27:21 <cooltshirtguy> didn't know that
18:27:23 <mowest> Is Fedora Server often in production or are most running it more as a test environment, homelab, check out new features... type of environments? Seems like CentOS Stream might be a better option for production, but I don't know if we have metrics on Fedora Server use cases.
18:27:28 <nirik> Ideally, every service would exit as fast as it can and a timeout wouldn't even be needed.
18:27:47 <pboy> Yes, that's the ideal world
18:27:55 <nirik> sadly we live in the real one. ;)
18:28:19 <cooltshirtguy> dont remind me
18:28:21 <cooltshirtguy> :)
18:28:29 <nirik> mowest: Personally I have both... production and test... but I don't know that we have a good sense of how much of each our userbase is.
18:28:29 <pboy> A technical question:
18:28:42 <eseyman> I would say our target audience is the homelab and SOHO
18:29:14 <pboy> A bit back:  our goal is to provide a reasonable  stable version for production
18:29:25 <pboy> It's in our PRD
18:29:26 <mowest> eseyman: That makes sense to me for our target audience.
18:29:54 <pboy> And yes, I have about 14 Fedora Server in Production
18:30:01 <mowest> I guess technically it is production in my homelab :-)
18:30:24 <mowest> It is the only server OS in my homelab.
18:30:27 <pboy> Back to my technical question:
18:30:45 <pboy> nirik, do you know where the mentioned 2 mins are configured?
18:30:59 <pboy> In the etc conf file I see 90 secs
18:31:05 <nirik> pboy: not fully sure, I think in the /etc/system/systemd.conf...
18:31:20 <pboy> And systemctl show on a F36 system says 90 secs, too
18:31:21 <nirik> huh.
18:31:34 <nirik> perhaps it is 90s then... and they havent landed the 45s change
18:32:02 <pboy> No, 90 secs is also mentioned in RHEL docs
18:32:36 <pboy> and iiis 36, prior to the change yesterday
18:33:25 <pboy> Well, obviously, we don't know at the moment?
18:34:19 <nirik> yeah, not sure, would have to dig around/ask people.
18:34:43 <pboy> It would be helpful, if. you could do that.
18:35:00 <pboy> So I guess, we can leave the topic for today?
18:35:24 <pboy> We do the work as discussed and continue nest meeting.
18:35:47 <pboy> OK, I switch
18:35:58 <pboy> #topic web site revamp
18:36:14 <mowest> Last week I attended the video conference of the Fedora Council that focused on the website redesign.
18:36:23 <mowest> I introduced myself to the website redesign team, and got connected with their Matrix and Gitlab issue tracker for the Server landing page.
18:36:31 <mowest> A designer from Red Hat was going to DM me some time soon and go over goals for our landing page.
18:36:39 <mowest> It is supposed to look something like this one for Workstation or CoreOS https://fedora.gitlab.io/websites-apps/fedora-websites/fedora-websites-3.0/workstation/
18:36:51 <mowest> https://fedora.gitlab.io/websites-apps/fedora-websites/fedora-websites-3.0/coreos/
18:36:58 <mowest> If you have suggestions for the page you can either add a comment here: https://gitlab.com/fedora/design/team/wwwfpo-2022/-/issues/23
18:37:04 <mowest> Or I would be interested in taking your thoughts to the team as well.
18:37:09 <mowest> I will try to keep you updated at future IRC meetings.
18:37:40 <pboy> mowest +1 !
18:37:52 <nirik> mowest: thanks for taking that on!
18:37:55 <cooltshirtguy> mowest++
18:37:55 <zodbot> cooltshirtguy: Karma for mowest changed to 1 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
18:38:36 <pboy> I dislike wording like The leading ....
18:38:43 <pboy> These are hollow phrases
18:39:20 <pboy> Otherwise I like the graphical design
18:39:37 <mowest> I'm looking forward to working with the website team, they seemed excited to hear from someone on the Server WG.
18:39:58 <mowest> pboy: thanks for those thoughts, I will try to keep such wording off of our page.
18:40:33 <pboy> Yeah, I think we should start to collect our central ideas, our messages and what is important.
18:40:51 <pboy> MAybe, we create a ticket for the contentß
18:41:00 <pboy> ß -> ?
18:41:16 <mowest> I would love to come to the website team with some central ideas in mind already. So yes, please collect ideas for our messaging.
18:41:55 <pboy> #action pboy to create a ticket to collect our central messages for the new web site.
18:42:50 <pboy> Anything else?
18:43:00 <mowest> No that is all I have at this time.
18:43:09 <pboy> OK. Thanks!
18:43:25 <pboy> #topic Using Ansible to install and configure Wildfly
18:43:43 <pboy> John had to leave, unfortunately.
18:44:11 <pboy> We had a discussion on our mailing list:
18:44:26 <pboy> #link https://lists.fedoraproject.org/archives/list/server@lists.fedoraproject.org/thread/HEHEXBYFVD6E6SG33SOGJBREAEBJHGLE/
18:45:22 <pboy> Jason, anything to discuss now or should we postpone until we are complete again?
18:46:21 <eseyman> can we get access to John's work anywhere?
18:46:23 <cooltshirtguy> john and I emailed back and forth but i hadnt chance to get back to it. hopefully this weekend or the coming week.
18:46:40 <cooltshirtguy> he has it on github but needs an update from him
18:46:49 <pboy> OK, so let's postpone it (unfortunately again).
18:47:00 <eseyman> FTR, I can no longer find it on github
18:47:13 <cooltshirtguy> let me dig to see if i can find
18:47:36 <eseyman> he just has a config files repo
18:47:48 <cooltshirtguy> https://gitlab.com/jwhimpel/wildfly-ansible
18:47:59 <cooltshirtguy> gitlanb sorry, wrong one
18:48:05 <cooltshirtguy> *gitlab
18:48:43 <pboy> Anything else?
18:48:44 <eseyman> ah, much better
18:49:43 <pboy> Well,  I switch unless ....
18:49:45 <pboy> 3
18:49:51 <pboy> 2
18:50:06 <pboy> 1
18:50:11 <pboy> #topic Server WG work program for upcomming year
18:50:30 <pboy> We had a good discussion about it last meeting.
18:50:45 <pboy> We we made a good plan.
18:50:52 <cooltshirtguy> good meeting
18:50:59 <pboy> Anything to add / to amend?
18:51:24 <eseyman> no, I'm good
18:51:41 <cooltshirtguy> I'm good
18:51:48 <pboy> Good, next one
18:52:00 <pboy> #topic Open floor
18:52:16 <pboy> The floor is open.
18:52:22 <pboy> Who is the first?
18:53:03 <mowest> This is probably controversal, but I would like to see our default install switch to btrfs.
18:53:12 <mowest> Like workstation.
18:53:31 <pboy> Yeah, that's controversal, indeed
18:53:49 <mowest> Btrfs continues to improve and has the potential to be on par with ZFS in the server world.
18:54:03 <mowest> Even Ubuntu has become more quiet about ZFS.
18:54:27 <pboy> BTRFS violates our principal of strict data separation
18:54:45 <mowest> My thought is that Fedora Server should be on some of the cutting edges of the tech stack.
18:54:51 <pboy> At least, if there is one huge unstructured BTRFS
18:55:10 <eseyman> I suspect we're going to need compelling reasons to switch from xfs
18:55:26 <pboy> eseyman I think so, too
18:55:35 <nirik> there's a few, but yeah... it's hard to move to something new there.
18:55:48 <mowest> Snapshots?
18:56:03 <nirik> compression, checksums on data...
18:56:04 <pboy> Snapshots are available with LVM as well
18:56:09 <mowest> Not sure if Workstation supports those well yet.
18:56:49 <cmurf> pboy: When I complained about this concept going in the, was it the tech spec or product requirements, I was told that this was not going to be used as an argument against Btrfs. And here we are.
18:57:10 <mowest> I know for my homelab my file server has its data pool in btrfs and the main root drive is still the default xfs.
18:57:39 <pboy> There was one argument with BTRFS, snapshot and backups . But the discussion showed, the availabler tools make no difference.
18:57:40 <nirik> I used btrfs on my new server at home here.
18:58:05 <cmurf> I think it's splitting hairs to say using btrees (subvolumes) to separate user data from system is not strict, but using GPT or LVM (partitions) is strict.
18:58:11 <pboy> cmurf: No, maybe we find a way or use BTRFS with hat principle.
18:58:51 <cmurf> All of it is software separation, unless you put data and system on separate drives.
18:59:05 <pboy> Yes, an I use BTFRS on our servers on a distinct partition.
18:59:36 <pboy> cmurf it is not just black and white
18:59:43 * nirik has to head to another meeting.
19:00:01 <pboy> nirik thank you a lot!
19:00:04 <eseyman> I'm of the opinion that we want logical separation of system and user data, not physical separation
19:00:41 <pboy> eseyman in BTRFS there is at best a weak logical separation.
19:01:14 <cmurf> I don't understand the argument why that matters at all.
19:01:34 <cmurf> It's all weak logical separation, including LVM, especially if it were thin provisioning because all of that is also done with btrees.
19:02:30 <pboy> cmurf the weakness is of different grade
19:02:33 <eseyman> we want to make it easier to snapshot/backup/restore one without impacting the other
19:03:16 <cmurf> I think for initial provisioning you might be better off with Btrfs, not knowing anything else, with the option for /var to be setup as XFS. Some of this might hinge on the anaconda GUI changes that are planned.
19:03:21 <eseyman> note that I'm not enough of a filesystem expert to determine which one is the better choice. I'm just stating a principal we find useful
19:03:22 <pboy> eseyman regarding the available tools there is nor difference using snapshotrs with BTRFS or with LVM
19:04:19 <pboy> cmurf Good idea, lets us discuss how we could use BTRFS in server along with xfs and different requirements
19:05:00 <cmurf> LVM thick provisioned snapshot performance is not great and requires a target created in advance of a specific size. LVM thin provisioning is quite a lot more performant, and doesn't require advance allocation - it'll use the entire thin pool. But you can't really argue that it's strict separation because it's btree based, not much different from Btrfs subvolumes.
19:05:01 <eseyman> is btrfs an existing option already ?
19:05:02 <pboy> On my server, I use xfs for system and databasers, btrfs for a lot of other data, as an example.
19:05:26 <cmurf> eseyman: yes
19:05:58 <cmurf> Custom partitioning, choose the Btrfs scheme in the partition pop-up menu, click auto partition text, done. Install. Works fine.
19:06:04 <pboy> eseyman We should have RHEL back in our head
19:06:36 <pboy> I see, our time is up.
19:06:40 <eseyman> pboy: I agree up to a point
19:06:48 <cmurf> I'm not sure RHEL should be a consideration because RH is going to do what they want based exclusively on their customers. And Fedora server working group should do what they want based on the community. They're completely different things.
19:07:25 <pboy> cmurf yes, to some degree. But we are kind of upstream
19:07:44 <cooltshirtguy> good conversation but I got to drop for another meeting
19:07:49 <eseyman> I'm really not hearing our community screaming they want btrfs by default
19:08:06 <cmurf> Sure but they routinely and without complain don't implement things the way Fedora does if it doesn't mesh with what their customers want.
19:08:36 <pboy> cmurf agreed.
19:08:50 <eseyman> I believe we're overstayed our welcome. Let's take it to the list
19:08:56 <cmurf> 😄
19:08:58 <pboy> Nevertheless, we should be coutious.
19:09:19 <pboy> OK. Bye Bye to everybody.!
19:09:30 <pboy> #endmeeting