14:01:29 <tflink> #startmeeting
14:01:29 <zodbot> Meeting started Mon Apr 20 14:01:29 2015 UTC.  The chair is tflink. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:01:29 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
14:01:36 <tflink> #meetingname fedora-qa-devel
14:01:36 <zodbot> The meeting name has been set to 'fedora-qa-devel'
14:01:40 <tflink> #topic role call
14:01:50 * mkrizek is here
14:01:58 * kparal 
14:02:08 <tflink> #chair kparal mkrizek
14:02:08 <zodbot> Current chairs: kparal mkrizek tflink
14:02:56 <tflink> no josef today?
14:03:08 <kparal> he's on his way
14:03:32 * jskladan1 is present
14:03:42 <tflink> #chair jskladan1
14:03:43 <zodbot> Current chairs: jskladan1 kparal mkrizek tflink
14:03:51 <tflink> cool, shall we get started?
14:03:59 <tflink> who wants to go first with status?
14:04:14 <mkrizek> I will
14:04:20 <mkrizek> #topic mkrizek status report
14:04:27 <mkrizek> #info works on remote execution for disposable clients
14:04:31 <mkrizek> #info now trying to make paramiko send stdout/stderr in realtime
14:04:37 <mkrizek> #info ETA is this week
14:05:05 <mkrizek> questions?
14:05:07 <tflink> sounds good, looking forward to the review
14:05:31 <mkrizek> that's about it from me
14:06:05 <tflink> next?
14:06:36 * kparal still writing his status
14:06:46 <kparal> :)
14:06:53 <tflink> jskladan1?
14:06:58 <jskladan1> #topic jskladan status update
14:06:59 <jskladan1> #info resultsdb's slowness was caused by the absence of indexes on ForeignKeys in Postgres - should be sorted out now
14:06:59 <jskladan1> #info OpenQA runs in Docker now - we might discuss replacing Suse on the Boston machine with fedora + containers
14:08:40 <tflink> glad that we appear to have the slow query thing figured out
14:08:49 * tflink forgets - did that get deployed to dev or stg?
14:09:24 <tflink> doesn't look like it :(
14:09:42 <tflink> #action tflink to release new resultsdb and deploy to dev/stg
14:09:44 <jskladan1> tflink: not sure, to be honest - we were talking some downtime, and I'm not sure how it ended
14:10:02 <kparal> I think it did, I haven't seen those exact errors in the last few days. but I overlooked them
14:10:21 <tflink> we got rid of the long queries in dev/stg, so they wouldn't show up
14:10:24 <jskladan1> kparal: that is most probably caused by the fact, that resultsdb does not return data
14:10:26 <kparal> the problem is that the debug log is truncated, so often we can't be sure
14:10:55 <tflink> yeah, I also want to get the new libtaskotron deployed to dev/stg today to get that tested
14:11:06 <jskladan1> kparal: ready yet? :)
14:11:09 <kparal> tflink: please wait for one more patch, coming today
14:11:11 <kparal> yes
14:11:20 <kparal> #info kparal status update
14:11:22 <tflink> kparal: ok, will do
14:11:23 <kparal> damn
14:11:26 <kparal> #topic kparal status update
14:11:45 <kparal> #info the last time I played with virt-builder (disposable clients feature) was three weeks ago, in the last two weeks I was swamped with manual testing and only had time to try to debug specific errors in our buildbot logs. for many of those I submitted patches that will hopefuly fix or improve them, e.g.:
14:11:49 <kparal> https://phab.qadevel.cloud.fedoraproject.org/D339
14:11:50 <kparal> https://phab.qadevel.cloud.fedoraproject.org/D336
14:11:50 <kparal> https://phab.qadevel.cloud.fedoraproject.org/D337
14:11:50 <kparal> https://phab.qadevel.cloud.fedoraproject.org/D323
14:11:53 <kparal> #info one more patch coming today for libtaskotron exceeding the buildot timeout when working with a lot of updates
14:12:31 <kparal> there's an avalanche of errors from buildbot, we need to do something about it. hopefully with the latest patches it will improve substantially
14:12:43 <tflink> yeah
14:12:52 <tflink> hopefully we'll have it figured out before freeze ends
14:13:08 <tflink> once the queues empty out, we hsould have fewer timeout issues either way
14:13:28 <kparal> I spent 4 hours today downloading packages to run a depcheck run and simulate some of those issues
14:13:48 <tflink> sounds like you found a new issue?
14:14:05 <kparal> it seems it is mostly caused by bodhi being slow and buildbot not detecting any new output on stdout for 40 minutes
14:14:17 <kparal> so I added some progress reports to stdout
14:14:36 <tflink> cool, sounds good
14:14:46 <kparal> originally I suspected yamlish issue returning, but all seems good
14:14:47 <tflink> do you think that the second patch will be ready today?
14:14:53 <kparal> tflink: yes
14:15:20 <kparal> I'm just stuck installing fake_fedorainfra to try posting to bodhi
14:15:32 <tflink> what's going wrong?
14:15:35 <kparal> but I'll try to make it work somehow
14:15:43 <kparal> not sure, I probably don't know how to configure it properly
14:16:37 <tflink> let me know if I can help - I thought it was documented enough to figure out reasonably but it sounds like not so much
14:16:37 <kparal> I wondered if we could deploy a public instance of fake_fedorainfra and resultsdb that anybody could use for dev purposes. and we would wipe the db regularly
14:16:47 <kparal> it would save people a lot of time :)
14:17:17 <kparal> tflink: actually, I haven't found any readme, but I haven't searched in the wiki yet
14:17:18 <tflink> or improve the docs for deploying them - I usually find that to be quicker
14:17:59 <tflink> we can look into it, though
14:18:29 <kparal> ok, we can discuss this further after the meeting
14:18:32 <kparal> that's all from me
14:18:40 <tflink> #topic tflink status update
14:18:51 <tflink> #info upgraded stg to f22, newest libtaskotron
14:18:51 <tflink> #info helped triage the timeout issue in resultsdb
14:18:51 <tflink> #info got disposable client skeleton pushed
14:18:51 <tflink> #info working on production taskotron re-deployment and prep this week
14:18:52 <tflink> #info working on more disposable client stuff (hopefully) this week
14:19:45 * tflink is planning to push a new version of libtaskotron as soon as the next patch is pushed
14:20:58 <tflink> any questions/comments?
14:21:18 <jskladan1> nope
14:21:24 <tflink> ok, moving on
14:21:33 <tflink> #topic upgrading taskotron production
14:22:00 <tflink> other than the timeout issues which are hitting dev, stg and prod - is anyone aware of any reasons not to upgrade production after freeze is over?
14:22:46 <jskladan1> nothing here
14:22:49 <jskladan1> maybe kparal?
14:22:59 <kparal> I'm all for upgrading production
14:23:01 <mkrizek> no reasons
14:23:19 <kparal> with the latest patches
14:23:22 <tflink> #info no known issues that would cause us not to upgrade production as planned after freeze is lifted
14:24:13 <tflink> freeze will be lifted tomorrow, I don't remember what time of day it's lifted but I was thinking of doing it either tomorrow, wednesday or thursday
14:24:58 <jskladan1> sounds good to me
14:25:25 <tflink> maybe wed-thurs would be better to get a bit more time to test out the new patches that will be released today
14:26:18 * tflink will send out announcements before actually doing it since this will involve downtime
14:26:48 <tflink> any other questions or comments?
14:27:02 <mkrizek> nope
14:27:11 <jskladan1> nn
14:27:16 <tflink> ok, moving on
14:27:21 <tflink> #topic beaker
14:27:38 <tflink> I found out last week that some new machines have been donated for use with beaker
14:28:02 <tflink> which kinda increases pressure to get something working as a production system instead of the current dev system
14:28:31 <tflink> spoke with the beaker devs last night and they've started working on an ansible patch to get that at least started
14:28:56 <tflink> but I suspect that they'll need help making everything work with infra
14:29:18 <tflink> the debate in my mind is how high of a priority do we make this?
14:29:39 <tflink> theoretically, the task to import new trees into beaker should be working
14:31:08 <tflink> any thoughts?
14:31:56 <kparal> taskotron infra is definitely of a higher priority, I would say
14:32:18 <kparal> I guess you're asking more about taskotron development vs beaker work
14:32:38 <tflink> yeah
14:34:15 <kparal> I think this effort should be a collaboration of several teams, our team can't do everything. so does it sound reasonable to say that we shouldn't be e.g. putting 10x times more time into it than beaker devs do?
14:34:39 <tflink> yeah, that makes sense for now
14:34:57 <tflink> hopefully we'll see the first patch in the next week or so and can move on from there
14:35:00 <kparal> it should be balanced, so I think even the times spent across different teams should be somewhat balanced
14:36:29 <tflink> in an ideal world, I'd agree :)
14:36:49 <tflink> at some point, we may need to decide how important having beaker in fedora is, but that doesn't need to be today
14:38:41 <tflink> it sounds like there aren't many more thoughts on this - mostly wanted to give a heads up and get thoughts started
14:38:56 <kparal> how much hardware do we have now?
14:39:00 <kparal> for beaker
14:39:24 <tflink> with the old boxes that are going away around october - 10 boxes, 2 aren't racked yet
14:40:02 <kparal> so 10 bare-metal clients
14:40:13 <kparal> ok
14:40:21 <tflink> bare metal and virthosts
14:40:30 <tflink> the clients that are actually in beaker are vms right now
14:40:45 <tflink> I don't think I ever got back to adding the bare metal clients after the firewall issues were fixed
14:41:26 <tflink> nope, doesn't look like it
14:42:29 <tflink> anyhow, will keep folks updated as things change
14:42:39 <tflink> it'd be nice to have a decision on beaker soon
14:42:49 <tflink> before october, I mean
14:43:23 <tflink> #topic tasks for the next week
14:43:33 <tflink> is anyone needing tasks for the next week?
14:44:18 <kparal> I don't, unfortunately
14:44:42 <kparal> mkrizek went to the bathroom, it seems
14:45:04 * jskladan1 also has enough on his plate ATM
14:45:36 <mkrizek> I have enough tasks to keep me busy for two weeks I think
14:45:48 <tflink> k, I suspect that I'll mostly be busy with infra stuff for the next several days
14:46:49 <tflink> which sounds like time for
14:46:53 <tflink> #topic open floor
14:46:59 <tflink> anything else that folks wanted to bring up?
14:48:03 * kparal nothing here
14:49:09 <mkrizek> nope
14:49:18 <tflink> ok, I'll send out minutes shortly
14:49:21 <tflink> thanks for coming, everyone
14:49:24 <kparal> thanks
14:49:26 <tflink> #endmeeting