15:00:40 <mkonecny> #startmeeting Infrastructure (2020-02-27)
15:00:40 <mkonecny> #meetingname infrastructure
15:00:40 <mkonecny> #chair nirik pingou relrod smooge tflink cverna mizdebsk mkonecny abompard bowlofeggs
15:00:40 <mkonecny> #info Agenda is at:
15:00:40 <zodbot> Meeting started Thu Feb 27 15:00:40 2020 UTC.
15:00:40 <zodbot> This meeting is logged and archived in a public location.
15:00:40 <zodbot> The chair is mkonecny. Information about MeetBot at
15:00:40 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:00:40 <zodbot> The meeting name has been set to 'infrastructure_(2020-02-27)'
15:00:40 <zodbot> The meeting name has been set to 'infrastructure'
15:00:40 <zodbot> Current chairs: abompard bowlofeggs cverna mizdebsk mkonecny nirik pingou relrod smooge tflink
15:00:40 <mkonecny> #topic aloha
15:00:50 <mkonecny> .hello zlopez
15:00:51 <zodbot> mkonecny: zlopez 'Michal Konečný' <>
15:00:53 <nirik> morning
15:00:55 <cverna> hello
15:01:28 <nirik> huh, the board isn't loading right for me here. oh well.
15:01:59 <mymindstorm> hi
15:02:09 <tflink> morning
15:02:16 <mkonecny> nirik: Which board?
15:02:32 <nirik>
15:02:44 <mkonecny> I lost the highlighting today
15:02:50 <mkonecny> Not sure why
15:03:01 <mkonecny> #topic Next chair
15:03:01 <mkonecny> #info magic eight ball says:
15:03:01 <mkonecny> #info 2020-02-27 - mkonecny
15:03:01 <mkonecny> #info 2020-03-05 - pingou
15:03:02 <mkonecny> #info 2020-03-12 - ???
15:03:21 <mkonecny> Next week it will be on pingou
15:03:30 <cverna> oh did clear the highlighting it was super difficult to read anything
15:03:39 <cverna> I*
15:03:45 <mkonecny> cverna: Ok, thanks for info
15:04:10 <mkonecny> Who wants to drive meeting on 2020-03-12
15:04:10 <mymindstorm> cverna: there is a setting to hide the colors w/o clearing them
15:04:17 <mkonecny> Any volunteer?
15:04:54 <cverna> mymindstorm: ah good to know :)
15:05:29 <mkonecny> It looks like nobody is volunteering, we could decide this next week, moving on
15:05:40 <mkonecny> #topic New folks introductions
15:05:40 <mkonecny> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
15:05:40 <mkonecny> #info Getting Started Guide:
15:05:46 * pingou passing by but cannot really join the meeting, sorry
15:06:18 * nirik waves at pingou on the ship drifting by
15:06:25 <mkonecny> Anyone new here?
15:06:27 * pingou waives back
15:07:43 <smooge> nirik, looks like some of the certs expired: Websites prove their identity via certificates, which are valid for a set time period. The certificate for expired on 25 February 2020.
15:08:03 <mkonecny> Nobody new it seems, next topic
15:08:04 <nirik> yeah, I got that too, but then in loaded. Might be some nodes are messed up and some are not.
15:08:06 <mkonecny> #topic announcements and information
15:08:06 <mkonecny> #info ops folks are doing a 30min ticket triage every day at 19UTC in #fedora-admin - please join
15:08:06 <mkonecny> #info Fedora Infrastructure will be moving in 2020-06 from its Phoenix Az datacenter to one near Herndon Va. A lot of planning will be involved on this. Please watch out for announcements on changes.
15:08:06 <mkonecny> #info Old openstack cloud is being decommissioned 2020-03-01. Working with instance owners to make sure everyone is migrated off.
15:08:08 <mkonecny> #info future of FMN (
15:08:35 <nirik> #info We are in F32 beta freeze! Make sure you submit for review any patches to frozen machines!
15:09:25 * nirik liked jcline's idea for a FMN replacement... wonder if it would be worth having a pair of folks try and put together a proof of concept in stg...
15:09:43 <mkonecny> #info CPE Sustaining team started doing daily standups at 12 UTC from Monday till Thursday
15:10:14 <cverna> nirik : yeah I think it is a good way forward, we need to find the pair of folks tho :P
15:10:24 <mkonecny> nirik: I didn't know he started working on something
15:10:28 <jcline> The code is definitely not feature complete for what I described and got done early on during fedora-messaging so some things it implements got moved to fedora-messaging
15:10:33 <nirik> and +1 on the 12UTC standups!
15:11:00 <nirik> mkonecny: basically use rabbitmq to do it all (or as much of it as possible)
15:11:20 <mkonecny> nirik: I think this is a good idea
15:11:33 <nirik> yeah, less custom apps is good.
15:11:41 <mkonecny> Less work for us
15:11:43 <cverna> less custom apps ++
15:12:41 <mkonecny> Do we have any other announcements?
15:13:25 <cverna> nope :)
15:13:56 <mkonecny> #topic Taskotron EOL date - tflink
15:13:56 <mkonecny> #info taskotron is not moving to the new datacenter and we'd like to figure out a more specific EOL date, what dates gel reasonably well with CPE plans?
15:14:21 <pingou> I should have news about this in the coming weeks
15:14:32 <tflink> that's pretty much it - we're trying to figure out an exact date for turning off taskotron
15:14:35 <pingou> (potentially as soon as next week)
15:14:48 <pingou> (and by it, I meant resultsdb)
15:15:04 <mkonecny> Ok, let get more info next week
15:15:10 <nirik> I think that would be good to know, but yeah, after that we can schedule soemthing
15:15:21 <tflink> i'd like to make an annoucement soon
15:15:35 * nirik nods.
15:15:37 <pingou> tflink: what do you have in mind currently?
15:15:38 <tflink> there was already some confusion at the FESCo meeting this week
15:16:25 <tflink> pingou: other than before the move, I'm flexible so long as we don't have to do another upgrade
15:16:51 <pingou> do we have anything on our sides that rely on taskotron?
15:17:02 <pingou> can't think of any atm
15:17:13 <tflink> not really, the checks were never required by anything
15:17:32 <tflink> there are people who look at the rpmdeplint results
15:17:37 <pingou> if so, I'd say pick a date :)
15:17:58 <tflink> ok, that works
15:18:14 <tflink> we'll figure out a date for turning off taskotron (sans resultsdb)
15:18:30 <tflink> and leave the resultsdb part of it until a new home is figured out
15:19:35 <mkonecny> #topic Oncall
15:19:35 <mkonecny> #info
15:19:35 <mkonecny> #info nirik is oncall 2020-02-13 -> 2020-02-20
15:19:36 <mkonecny> #info cverna is oncall 2020-02-20 -> 2020-02-27
15:19:36 <mkonecny> #info pingou is oncall 2020-02-28 -> 2020-03-05
15:19:38 <mkonecny> #info ??? is oncall 2020-03-06 -> 2020-03-12
15:20:06 <nirik> I can take the next one I guess... unless someone else wants it. ;)
15:20:16 <nirik> (the 03-06 to 03-12)
15:20:35 <mkonecny> It's yours
15:20:39 <mkonecny> #info nirik is oncall 2020-03-06 -> 2020-03-12
15:21:04 <mkonecny> #info Summary of last week: (from current oncall )
15:21:52 <mkonecny> cverna: This is on you
15:22:17 <cverna> not much, I think I had one ping about accessing the testday box in fedorainfracloud
15:22:38 <cverna> but smooge helped with that
15:22:44 <cverna> thanks smooge++
15:22:58 <nirik> yeah... people are starting to use tickets more I think...
15:23:03 <mkonecny> It looks like the tickets are getting filled instead
15:23:13 <cverna> yes :)
15:23:13 <nirik> related, I'd love to get our 'how to work with the team' doc done and advertised.
15:23:22 <pingou> +1
15:23:41 <mkonecny> nirik: I think it's done and published, but not visible on docs.fp.o
15:24:14 <pingou> it is visible:
15:24:21 <pingou> there is just no links to it
15:24:40 <cverna> we could advertise it then a bit more
15:24:41 <mkonecny> Ok, I meant it's missing link on homepage
15:24:46 <pingou> yup
15:25:01 <cverna>
15:25:01 <nirik> I was thinking adding it to oncall ping reply...
15:25:06 <mkonecny> What about adding it to the startmeeting stuff?
15:25:14 <nirik> and perhaps /topics ?
15:25:22 <cverna> +1 to all of these :)
15:25:53 <nirik> someone want to file a ticket on it? :)
15:26:05 <mkonecny> We should start thread about this in our mailing list to get more ideas
15:26:35 <mkonecny> Or maybe in some other list, so we know where the people will look for it
15:27:12 <nirik> sure, +1.
15:27:17 <nirik> someone want to start it?
15:27:34 <mkonecny> I will send the mail after the meeting to our list
15:27:51 <mkonecny> Just adding this to my TODO list
15:28:46 <mkonecny> Ok, I'm moving to next topic
15:28:48 <nirik> +1
15:28:50 <mkonecny> #topic Monitoring discussion [nirik]
15:28:50 <mkonecny> #info
15:28:50 <mkonecny> #info Go over existing out items and fix
15:28:57 <nirik> ok, lets see.
15:29:12 <nirik> we had a lot of trouble with nagios the last few days due to moving some things to aws
15:29:34 <cverna> koji seems to have an episode
15:29:37 <nirik> 3 down hosts currently should be not there.
15:30:10 <nirik> yeah, I saw the alerts, not sure whats going on there.
15:30:18 <nirik> all the rest is stuff we have gone over before
15:31:35 <mkonecny> Moving to next topic than
15:31:36 <mkonecny> #topic Tickets discussion [nirik]
15:31:36 <mkonecny> #info
15:32:05 <nirik> nothing marker meeting.
15:32:30 <nirik> anything anyone wants to discuss ticket wise?
15:33:00 <nirik> we talked a bit about prioritizing backlog
15:33:11 <mkonecny> #topic backlog discussion
15:33:11 <mkonecny> #info go over our backlog and discuss and determine priority
15:33:36 <cverna> yes do we want to introduce the yummy vs trouble index :)
15:33:48 <cverna> I am reusing nirik terms :)
15:33:59 <nirik> it's a easy one to understand
15:34:02 <mkonecny> cverna: I think it is the best metric for this
15:34:14 <nirik> and how do we mark things? tags are messy, there's too many
15:34:18 <nirik> custom field?
15:34:31 <cverna> +1 so we can filter on them
15:34:42 <nirik> can you filter on custom?
15:34:44 * nirik looks
15:34:53 <cverna> ha maybe not :P
15:35:18 <mkonecny> I can recommend the prioritization matrix for this, it's using similar metrics
15:35:41 <cverna> mkonecny: +!
15:35:47 <cverna> rhh +1 even
15:35:48 <cverna> :)
15:36:12 <pingou> I think we can filter on custom field using field:foo in the search
15:36:14 <mkonecny> We need more ! :-D
15:36:14 <nirik> yeah, I meant to read that, but a skim sounds good
15:37:54 <nirik> I would definitely like to filter on it
15:39:07 <cverna> yes would be good
15:39:24 <cverna> so that we can get all the yummy without much trouble done first :)
15:39:24 <nirik> perhaps we should discuss more on list?
15:39:42 <nirik> and then we may need some special sessions to mark things.
15:40:20 <cverna> I can start a thread on the list
15:40:57 <mkonecny> nirik: I think it will be best to choose only few tickets and prioritize them and then go with next batch
15:41:17 <nirik> mkonecny: yeah, that makes some sense...
15:41:32 <nirik> mmight make sense to start oldest ?
15:41:59 <cverna> sounds good
15:42:16 <nirik> or we could do a few a week in this meeting if we can get a good process
15:42:47 <mkonecny> It will be maybe good to create some document, where everybody could vote for yummy vs. trouble and then you could determine the priority
15:43:23 <cverna> oh that sounds like it will be hard to maintain long term
15:43:29 <mkonecny> This will make the whole process asynchronous without any necessary meeting
15:43:46 <cverna> ha I see what you mean
15:44:01 <cverna> a weekly email on the list ?
15:44:02 <mkonecny> cverna: You could have only few tickets in it at one time and then clear it and add next batch
15:44:33 <cverna> then async discussion could take forever :(
15:44:33 <mkonecny> Yes, weekly email sounds good
15:44:49 <mkonecny> One batch - one week
15:44:56 <cverna> I don't have a strong opinion on that
15:44:59 <nirik> possibly. I like async
15:45:04 <mkonecny> How much tickets we have in the backlog?
15:45:18 <cverna> in infra 120 I think
15:45:20 <nirik> well, there's a number of buckets.
15:45:25 <nirik> about 120 in infra
15:45:28 <cverna> lol :)
15:45:29 <nirik> a bunch of releng
15:45:33 <mkonecny> So twenty for each batch is not that much
15:45:34 <nirik> bugzilla bugs
15:45:50 <cverna> let's try the weekly email and see how that goes
15:46:05 <cverna> if that's does not work well we can try something else
15:46:09 <cverna> ?
15:46:09 <nirik> +1
15:46:14 <mkonecny> +1
15:46:22 <nirik> I think we need to try things more
15:46:31 <nirik> if it doesn't work, we can change
15:46:32 <pingou> try, evaluate, reconsider, this sounds awfully like agile
15:46:39 * pingou ducks
15:46:59 <pingou> \_o<
15:46:59 <mkonecny> pingou: We should be more agile :-)
15:47:11 * pingou called afk
15:47:15 <nirik> don't be afraid of failure. ;)
15:47:23 <cverna> I am stretching everday to increase my agility
15:47:42 <mkonecny> good for you
15:47:45 <nirik> I stepped over some logs the other day here... very agile!
15:47:57 <cverna> :D
15:48:15 <cverna> I am happy to start sending this emails
15:48:19 <mkonecny> Ok, we have one last topic
15:48:21 <mkonecny> #topic Open Floor
15:48:51 <cverna> nothing here
15:49:12 <mkonecny> Nothing to see here, move on
15:49:19 <nirik> nothing here either off hand
15:49:57 <mkonecny> Than I will end this productive meeting :-)
15:49:59 <mkonecny> #endmeeting