13:01:26 <mvollmer> #startmeeting 13:01:26 <zodbot> Meeting started Mon Aug 17 13:01:26 2015 UTC. The chair is mvollmer. Information about MeetBot at http://wiki.debian.org/MeetBot. 13:01:26 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 13:01:31 <mvollmer> .hello mvo 13:01:32 <zodbot> mvollmer: mvo 'Marius Vollmer' <marius.vollmer@gmail.com> 13:01:40 <mvollmer> #topic Agenda 13:02:00 <mvollmer> * Multipath 13:02:12 <stefw> * journalctl regression 13:02:24 <stefw> * Further testing fires 13:03:15 <mvollmer> * hubbot disk usage 13:03:33 <dperpeet> .hello dperpeet 13:03:36 <zodbot> dperpeet: dperpeet 'Dominik Perpeet' <dperpeet@redhat.com> 13:03:46 <stefw> .hello stefw 13:03:47 <zodbot> stefw: stefw 'Stef Walter' <stefw@redhat.com> 13:03:59 <dperpeet> * Makefile conventions 13:04:07 <petervo> dperpeet, rpmquery is available 13:04:10 <stefw> * Internet Explorer 11 13:04:12 <dperpeet> petervo, thanks! 13:04:43 <mvollmer> * Moving things out of the shell 13:04:55 <stefw> * Moving things out of cockpit-wrapper 13:06:38 <mvollmer> ok, let's start 13:06:45 <mvollmer> #topic Multipath 13:06:52 <mvollmer> So I have started playing with this 13:07:09 <mvollmer> some notes here: https://github.com/cockpit-project/cockpit/wiki/Storage---Multipath 13:07:17 <github> [cockpit] stefwalter opened pull request #2590: Update cockpit.spec for changes from RHEL (master...rhel-spec-updates) http://git.io/v3pmW 13:07:46 <mvollmer> should be almost straightforward to get Cockpit "multipath safe" 13:07:59 <mvollmer> more work to make it "multipath nice" 13:08:07 <stefw> nice research 13:08:14 <stefw> how much of the work should go into storaged? 13:08:17 <github> [cockpit] dperpeet pushed 1 new commit to master: http://git.io/v3pmd 13:08:17 <github> cockpit/master 967318a Marius Vollmer: test: Only skip check-journal with systemd 219-21.... 13:08:23 <mvollmer> i think we should offer UI for setting multipath up and configuring it 13:08:38 <mvollmer> storaged should expose the mpath configuration 13:08:45 <mvollmer> basically the multipathd state 13:08:46 <stefw> makes sense 13:09:08 <mvollmer> that is, storaged would need to wrap a d-bus api around multipathd 13:09:37 <mvollmer> the first bit of info we need is to identitfy which of the multiple block devices is the 'leader' 13:10:12 <stefw> phatina, have you seen the research mvollmer has done on multipath? 13:10:13 <mvollmer> initially, I'll do some regexp matching to find the device mapper device, but storaged should do that more directly 13:11:02 <mvollmer> I also found a bug in Cockpit re a misunderstanding of the storaged Drive property. 13:11:21 <phatina> stefw: yes 13:12:32 <mvollmer> phatina, I just learned about the sysfs slaves/ and holders/ construct 13:12:46 <mvollmer> maybe storaged could expose that? 13:13:06 <mvollmer> but I tink storaged should also natively know about multipath 13:13:20 <mvollmer> and then we don't need that generic slaves/ holders/ thing 13:13:52 <mvollmer> phatina, https://www.redhat.com/archives/dm-devel/2006-February/msg00091.html 13:15:06 <mvollmer> so anyway, I continue playing with this to learn more about which pieces can be relied upon. 13:16:14 <mvollmer> next topic? 13:16:29 <stefw> yip 13:16:45 <mvollmer> #topic journalctl regression 13:17:27 <stefw> A regression was committed up stream 13:17:28 <stefw> to systemd 13:17:33 <stefw> where journalctl -f -t unmatched 13:17:34 <stefw> no longer works 13:17:39 <dperpeet> 219-21 13:17:47 <stefw> and that patch was backported to 219-21 13:17:51 <stefw> and then pushed through f22 updates 13:18:00 <stefw> systemd git master now contains my fix to patch this 13:18:05 <stefw> but it's not yet fixed in F22 13:18:21 <stefw> https://bugzilla.redhat.com/show_bug.cgi?id=1253649 13:18:32 <stefw> Tests should be failing with this 13:18:41 <stefw> but we've shut off journal tests 13:18:46 <stefw> when that systemd version is present 13:19:02 <stefw> we need to remove the workaround once that bug is fixed 13:19:35 <stefw> that's it 13:19:37 <mvollmer> http://files.cockpit-project.org/hubbot/bug_status.html 13:19:44 <mvollmer> should be listed there ^^ 13:19:59 <mvollmer> cockpit itself is also broken by the regression, of course. 13:20:06 <stefw> yup 13:20:23 <mvollmer> but not completely: as soon as the display is not empty, it will work 13:20:33 <mvollmer> i hope that is a rare thing 13:20:52 <dperpeet> should we extend the tests? 13:20:55 <dperpeet> and catch the issue 13:21:03 <dperpeet> / handle it 13:21:04 <mvollmer> well, we do... :-) 13:21:28 <mvollmer> ahh, you mean workaround in the cockpit code? 13:21:37 <dperpeet> yeah 13:21:41 <mvollmer> detect a non-blocking journalctl? 13:21:45 <dperpeet> I don't think it's that necessary 13:21:47 <mvollmer> I'd say no. 13:22:04 <mvollmer> what would we do, poll? 13:22:19 <stefw> we certainly shouldn't fix our journal code to do that 13:22:29 <dperpeet> there's no practical way 13:22:33 <mvollmer> we should only do it when f22 drags their feet too long 13:22:43 <stefw> we should raise it with FESCO if they do 13:22:47 <stefw> i'll prepare a patch to the rpm build 13:22:51 <mvollmer> yeah 13:22:57 <dperpeet> that makes more sense 13:23:08 <mvollmer> why did they include the patch in the first place? 13:23:25 <mvollmer> probably to fix the bug that I have reported way back... :-) 13:24:21 <mvollmer> https://bugs.freedesktop.org/show_bug.cgi?id=71548 13:25:09 <mvollmer> next topic? 13:25:39 <mvollmer> #topic Further testing fires 13:26:02 <mvollmer> phantomjs is quite often crashing with check-login, no? 13:26:11 <stefw> yeah it's the first time i've s tarted seeing that 13:26:15 <stefw> was it due to disk space issues? 13:26:26 <mvollmer> right, might be 13:26:42 <dperpeet> we can start a blanket retest of failed tests tonight 13:26:46 <mvollmer> /home was 100% full. 13:26:49 <dperpeet> when it doesn't block that much 13:27:14 <dperpeet> this was due to a reluctance to automatically delete images 13:27:23 <dperpeet> mvollmer and I fixed this today 13:27:24 <mvollmer> yeah, my fault 13:27:38 <dperpeet> hubbot now cleans up instead of pushing things into the basement 13:28:02 <mvollmer> we still use about 120 GiB 13:28:09 <dperpeet> how much space do we have left? 13:28:23 <mvollmer> let me check 13:28:26 <dperpeet> daily master checks are a pretty significant drain 13:28:34 <mvollmer> /dev/mapper/vg_dhcpclient06-lv_home 204G 137G 57G 71% /home 13:28:39 <dperpeet> we could automatically delete those as soon as a test finishes 13:28:45 <mvollmer> heh, not so useful 13:28:50 <dperpeet> indeed 13:28:52 <mvollmer> 57G available 13:29:34 <dperpeet> we haven't changed the wildcard regarding images to keep yet 13:29:44 <dperpeet> right now I believe all of a pull requests images are kept 13:29:55 <dperpeet> while we only really need the newest one 13:29:58 <mvollmer> 36G cockpit-data 13:30:05 <mvollmer> 46G hubbot 13:30:12 <mvollmer> not as bad as I thought 13:30:37 <mvollmer> 9.5G /srv/testdata/ 13:30:37 <dperpeet> we can tweak this a bit 13:30:59 <dperpeet> or even change behavior to more aggressively prune when disk space is running low 13:31:18 <mvollmer> one idea is to only keep the last run for a given commit 13:31:31 <mvollmer> now we keep them all, mostly because lazy 13:31:55 <mvollmer> anyway 13:31:57 <dperpeet> we probably need to do both 13:31:58 <mvollmer> other fires? 13:32:02 <stefw> i'll also try to look at more fires, and fix them 13:32:04 <stefw> there's this one: 13:32:12 <stefw> https://github.com/cockpit-project/cockpit/issues/2591 13:32:22 <stefw> the thing that keeps me going is that so many of these are real bugs 13:32:25 <stefw> both in cockpit and elsewhere 13:33:16 <mvollmer> yep 13:33:42 <stefw> mvollmer, can i assign that one to you? 13:33:48 <mvollmer> yes 13:34:21 <stefw> dperpeet, have you discovered anything about Travis? 13:34:30 <stefw> is it time to move away from Travis if it keeps falling over? 13:34:33 <stefw> or should we disable our root tests 13:34:35 <dperpeet> the tests seem to work again 13:34:51 <dperpeet> the thing is, I have the feeling they want to move away from the type of test we use 13:34:52 <stefw> but i still see broken output: 13:34:52 <stefw> https://travis-ci.org/cockpit-project/cockpit/jobs/75573396 13:34:54 <dperpeet> towards non-root containers 13:35:00 <stefw> i can move it to non-root 13:35:03 <stefw> but it means disabling certain tests 13:35:06 <stefw> that never get run anywhere 13:35:21 <dperpeet> all the tests I started after last friday afternoon seemed fine 13:35:30 <stefw> what about this one? 13:35:30 <stefw> https://travis-ci.org/cockpit-project/cockpit/jobs/75573396 13:35:39 <stefw> i can restart it 13:36:07 <dperpeet> that's three days ago 13:36:09 <dperpeet> I think 13:36:12 <dperpeet> from the date 13:36:31 <dperpeet> mvollmer, what was the other ci system you linked? 13:36:41 <mvollmer> semaphoreci.com 13:37:09 <dperpeet> they have caching 13:37:12 <stefw> from one non-open-source to another 13:37:13 <stefw> wheeee 13:37:51 <dperpeet> I can test how much effort it would be to move to travis' new structure 13:37:57 <mvollmer> why do we use travis in the first place? 13:38:03 <stefw> because it was so easy 13:38:11 <mvollmer> could we extend hubbot to do the job now? 13:38:27 <dperpeet> it's just make check et cetera 13:38:30 <dperpeet> we could have a dedicated vm 13:38:40 <stefw> and you've invented jenkins 13:38:45 <stefw> well already done that with hubbot 13:38:46 <dperpeet> adds load to our server 13:39:04 <dperpeet> a distinct service adds variety to the testing 13:39:25 <dperpeet> how about I see how much effort it would be to get our tests to run in the new travis? 13:39:32 <dperpeet> with containers 13:39:54 <dperpeet> they require non-root but supposedly cache better 13:40:06 <stefw> it's just one line change 13:40:18 <stefw> i can make that change 13:40:20 <stefw> not a problem 13:40:24 <stefw> well two lines 13:40:32 <dperpeet> unless something doesn't work :) 13:40:40 <dperpeet> well, go for it 13:40:48 <dperpeet> if something crops up, we can start putting out the fires 13:42:30 <stefw> hmmm, no it's not thath easy 13:42:37 <stefw> "If you require sudo, for instance to install Ubuntu packages, a workaround is to use precompiled binaries, uploading them to S3 and downloading them as part of your build, installing them into a non-root directory." 13:42:56 <dperpeet> like I said, non-root 13:43:27 <mvollmer> is running the unit tests in mock good enough? 13:43:37 <mvollmer> the output is hidden, but we can polish that up 13:44:02 <stefw> 'make distcheck' 13:44:05 <stefw> 'make check-memory' 13:44:09 <dperpeet> in mock on travis or with hubbot? 13:44:13 <stefw> and a full autogen.sh 13:44:14 <mvollmer> hubbot 13:44:17 <stefw> all those things need to be tested 13:44:23 <stefw> to have a similar feature set 13:44:33 <dperpeet> we could have a vm and do what travis does 13:44:45 <dperpeet> except we cache the updated base system 13:44:49 <mvollmer> also clang builds 13:45:15 <mvollmer> but I'd say we didn't get much value from those, or did we? 13:45:16 <stefw> yup 13:45:25 <stefw> well often one would fail and not the other 13:45:30 <dperpeet> we'd be putting more eggs into the hubbot basket 13:45:45 <dperpeet> and load on the server 13:46:18 <mvollmer> ok, time for the next topic? 13:46:29 <dperpeet> one last question 13:46:38 <mvollmer> should we arrange more discussion about this? 13:46:39 <dperpeet> stefw, do you want to look into travis changes? 13:46:45 <dperpeet> just to be clear on the action 13:46:47 <stefw> yes doing that now 13:46:50 <dperpeet> ok 13:46:55 <dperpeet> /eot 13:47:16 <mvollmer> #topic hubbot disk usage 13:47:26 <mvollmer> we have covered that 13:47:37 <mvollmer> #topic Makefile conventions 13:47:54 <mvollmer> dperpeet? 13:47:55 <dperpeet> background on this is that we have quite a few variables in the Makefile.am files 13:48:00 <dperpeet> for the individual packages 13:48:06 <dperpeet> and it's not that easy to write or review 13:48:15 <dperpeet> to see if files are missing, and which file goes into what 13:48:33 <dperpeet> do we want to simplify this or somehow write up a mini-guideline? 13:48:40 <dperpeet> or do we even have one and I missed it 13:48:51 <dperpeet> with debugdata, gz, min, data, ... 13:48:57 <dperpeet> bundle 13:49:13 <stefw> we need a guideline 13:49:20 <stefw> unfortunately it seems hard to make it simpler than it is 13:49:22 * stefw cries 13:50:02 <dperpeet> maybe we can add a sanity check 13:50:13 <dperpeet> compare with files present 13:50:18 <dperpeet> or even write one if necessary 13:50:27 <dperpeet> and those files need to be in the variables somehow 13:50:45 <dperpeet> we can agree on writing it into the wiki or hacking.md 13:50:56 <github> [cockpit] stefwalter opened pull request #2592: Use Travis container based infrastructure (master...travis-container-based) http://git.io/v3puJ 13:51:10 <mvollmer> I guess I got some details wrong/forgot some file and that is what broke Travis 13:51:48 <stefw> well because travis runs 'make distcheck' 13:52:01 <dperpeet> that passed for me locally 13:52:05 <mvollmer> for me too 13:52:39 <stefw> also difference between --enable-debug 13:52:40 <stefw> and not 13:52:43 <stefw> so it's good to have these checks 13:52:56 <dperpeet> enable-debug is probably the culprit 13:53:21 <dperpeet> of course it is - I can start writing the guidelines 13:53:40 <dperpeet> #action dperpeet to start wiki page on Makefile.am conventions for packages 13:54:20 <mvollmer> next? 13:54:47 <mvollmer> #topic Internet Explorer 11 13:54:56 * mvollmer rushes a bit, sorry 13:55:15 <dperpeet> I was able to reproduce the error message we saw in https://github.com/cockpit-project/cockpit/pull/2566 13:55:29 <dperpeet> and I couldn't see it locally after a patch, waiting on tests to run 13:56:05 <dperpeet> functionality seems fine 13:56:18 <dperpeet> so after merging current cockpit should work fine with IE11 13:56:21 <dperpeet> /eot 13:57:03 <mvollmer> # Moving things out of the shell / out of the cockpit wrapper 13:57:13 <mvollmer> i did some mechanical work there 13:57:19 <mvollmer> lots more still to do 13:57:38 <mvollmer> one goal could be to first get the shell into a clean state 13:57:48 <mvollmer> and move the legacy stuff somewhere else 13:58:01 <mvollmer> and then continue to clean this up, as we feel like it 13:58:33 <dperpeet> I think refactoring is more important than actual cleanup at this point 13:58:53 <dperpeet> a clean structure makes it easier to clean the smaller parts as we touch them 13:59:26 <stefw> moving stuff out of cockpit-wrapper is important 13:59:29 <stefw> as it lets us run as a container 14:00:43 <mvollmer> yep 14:01:01 <dperpeet> I like the level of change you put into it so far 14:01:02 <mvollmer> ok, my plan is to return to this after multipath 14:03:17 <stefw> sounds good 14:03:37 <mvollmer> with priority on cockpit-wrapper 14:04:04 <dperpeet> yeah 14:04:12 <mvollmer> #topic Any other biz 14:05:10 <stefw> we need to do a release today 14:05:19 <stefw> everything marked priority is targetted to go in 14:05:55 <mvollmer> ok, I'll do https://github.com/cockpit-project/cockpit/pull/2589 14:06:13 <stefw> subin, when #2572 is ready, i can review again 14:06:23 <stefw> and i'll try to merge #2566 when it's done testing 14:06:25 <petervo> mvollmer, i'm in the process of doing that 14:06:36 <subin> stefw , working on it 14:06:38 <mvollmer> oh, sorry. 14:06:41 <stefw> great 14:07:59 <dperpeet> stefw, the fixup goes into the second commit (fix race condition) 14:08:02 <dperpeet> in #2566 14:08:09 <stefw> ok 14:08:15 <dperpeet> if it works :) 14:08:28 <mvollmer> ok 14:08:32 <mvollmer> #endmeeting