14:01:41 <pbrobinson> #startmeeting Fedora IoT Working Group Meeting
14:01:41 <zodbot> Meeting started Wed Nov 21 14:01:41 2018 UTC.
14:01:41 <zodbot> This meeting is logged and archived in a public location.
14:01:41 <zodbot> The chair is pbrobinson. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:01:41 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
14:01:41 <zodbot> The meeting name has been set to 'fedora_iot_working_group_meeting'
14:01:42 <pbrobinson> #chair pwhalen pbrobinson bcotton
14:01:42 <zodbot> Current chairs: bcotton pbrobinson pwhalen
14:01:42 <pbrobinson> #topic roll call
14:01:50 <lorbus> .hello2
14:01:52 <zodbot> lorbus: lorbus 'Christian Glombek' <cglombek@redhat.com>
14:01:55 <pbrobinson> morning, who's here, sorry I'm late
14:02:01 * pwhalen is here
14:02:05 <lorbus> o/
14:02:16 <pwhalen> good morning/afternoon/evening folks
14:03:06 <pbrobinson> I suspect we'll be reduced this week due to Thanksgiving tomorrow in the USA
14:03:52 <lorbus> I missed last week's mtg but I couldn't find its minutes, it did happen right?
14:04:19 <pwhalen> lorbus, no meeting last week
14:04:27 <lorbus> ahh ok. nvmd then
14:05:06 <pbrobinson> I was travelling and in face to face meetings
14:05:50 <pwhalen> is the US off both Thursday and Friday this week?
14:06:18 <pbrobinson> basically
14:06:25 <pbrobinson> and a lot have taken the whole week
14:07:25 <pbrobinson> right, so I think we'll keep it simple this week and focus on just a couple of things
14:08:35 <pbrobinson> #topic 1) ==== Fedora 29 blocker status for IoT ====
14:11:09 <pwhalen> so last week, in lieu of the meeting I sent out a list of current blockers
14:11:33 <pwhalen> #info initial-setup hangs during boot after IoT installation
14:11:34 <pwhalen> #link https://bugzilla.redhat.com/show_bug.cgi?id=1646568
14:11:55 <pwhalen> this looks to be fixed, (thanks mkolman!)
14:12:09 <pbrobinson> so there was a new compose yesterday that should fix the inital-setup bits
14:12:13 <pwhalen> I just need to do another install to confirm
14:12:14 <pwhalen> right
14:12:38 <pwhalen> it looks good on an install I did yesterday, no issues on boot
14:13:10 <pwhalen> be great if others can also take a look and confirm
14:13:27 <pbrobinson> so it also pulls in the 4.19.2-301 kernel which should fix the RPi3 WiFI
14:14:32 <pwhalen> if you do confirm the initial-setup issues, you can leave karma as well
14:14:38 <pwhalen> #info https://bodhi.fedoraproject.org/updates/FEDORA-2018-f9bf4608fa
14:15:16 <mtimc> should the wifi come up by default - I did an `rpm-ostree upgrade` today, I have a 4.19.2-301 kernel, and still have no wifi
14:16:07 <pwhalen> mtimc, does the device should up at all?
14:16:07 <pbrobinson> we've recently also added the intel wifi firmware which was missing due to an oversight
14:16:27 <mtimc> that depends on what 'should up' means ;-)
14:16:29 <pwhalen> s/should/show
14:16:33 <pwhalen> heh :)
14:16:40 <pbrobinson> mtimc: yes, in theory, there appears to be a separate unrelated wifi bug
14:16:46 <mtimc> ifconfig shows lo and eth0 only
14:17:17 * jsmith arrives late
14:17:58 * jsmith has a Raspberry Pi 3B+ on its way to help test
14:17:59 <pbrobinson> hey jsmith
14:19:41 <pwhalen> ok, next issue?
14:20:07 <pbrobinson> mtimc: pwhalen: basically I think that issue is similar or the same as seen on ARMv7, it would be useful if mtimc could file a bug on it so we can track
14:20:11 <pbrobinson> it's weird
14:20:32 <mtimc> I can do that. What's the expected behaviour?
14:20:39 <pbrobinson> next issue is the dracut/network time out when no wired ethernet is connected
14:20:54 <pbrobinson> I have a theory here but I need to tast it later today
14:21:29 <pwhalen> #info The 35network-legacy is still used even if network-scripts isn't installed in Fedora 29
14:21:30 <pwhalen> #link https://github.com/dracutdevs/dracut/issues/488
14:21:39 <pwhalen> for the logs
14:21:41 <pbrobinson> mtimc: expected behaviour is to see wlan0 device, what seems to happen is there wifi just isn't there "dmesg |grep brcm" only show BT bits
14:22:16 <mtimc> @probinson: ta. I'll create the bug
14:23:17 <mtimc> fwiw, the only brmc dmesg's relate to bluetooth
14:23:43 <pbrobinson> right, the other bug we've seen is issues around rpm-ostree upgrade/install https://bugzilla.redhat.com/show_bug.cgi?id=1648112
14:24:28 <mtimc> that's mine. sorry about the messy report.
14:24:35 <pbrobinson> this one seems RPi specific and even possibly specific models of SD card. It might be improved with the 4.19.x kernel but over all it's hard to reproduce
14:25:16 <pbrobinson> and it seems very specific to the Raspberry Pi, I'm not sure it's something we're going to be able to easily fix so I'm not sure it's something we can block a release on
14:25:17 <pwhalen> and it eventually works.
14:25:29 <pwhalen> pbrobinson, agreed
14:26:16 <mtimc> we've tried several pi3's from element 14 and several SDCard types. I've not tried a 3b+ yet
14:27:35 <ipcloud> I have the RPi3B+ and will test today
14:27:51 <mtimc> has anyone else seen it at all?
14:27:58 <pwhalen> I've also tried using an SSD, still hit it on upgrade
14:28:31 <mtimc> @pwhalen - it's a relief to hear that it's not just something that I'm doing
14:28:56 <pwhalen> mtimc, oh no, definitely seeing it as well
14:29:14 <pbrobinson> I have no doubt it's an issue the problem I have is that if we block the release we could wait for months tog get it fixed
14:29:37 <pbrobinson> and that's not something I'm prepared to do
14:30:26 <pbrobinson> ultimately the plan it to do regular releases every month to six weeks
14:30:40 <pwhalen> right, and it does work. I have a rpi that I ssh to and I have yet to see it there
14:31:22 <mtimc> that seems fine. there are some rpm's that I've never managed to install due to the error. That could be not pushing hard enough/trying enogh times
14:31:41 <pbrobinson> it might well be improved on 4.19 which landed in yesterday's compose, so I'd like to see people actually able to reproduce to try again with a clean 4.19 compose
14:31:42 <mtimc> pwhalen, what's different about that pi?
14:31:53 <pwhalen> mtimc, if you reboot, or just try again I find it often works. Very frustrating though
14:32:19 <pwhalen> mtimc, I havent been able to work that out (why I dont see it there)
14:32:51 <pbrobinson> Of the 4 or so devices I have it running on a 3B+ and I have yet to see it there, I need to put it on an original 3, and find some different cards, I just need time
14:33:16 <mtimc> it is a very slow and manual testing process
14:34:31 <pwhalen> mtimc, indeed
14:34:49 <pbrobinson> mtimc: btw I saw no wifi on my 3B+, powered it off, replugged power, booted and the wifi is now there.
14:35:13 <mtimc> is there any way to get a clean compose that's 'ready to go' - avoiding the setting up user, etc - I can then put pi's in test harnesses
14:35:31 <pbrobinson> mtimc: in terms of testing we're working to get a CI/CD system so we can do a level of automated testing
14:36:00 <pbrobinson> mtimc: you should be able to do that with a kickstart
14:36:47 <mtimc> pbrobinson, where's the pre-kickstart image? I'm only aware of the images that need the arm-installer
14:37:41 <mtimc> does koan+cobbler work for this type of setup?  I used to use them for vm creation, but koan seemed to start to rot.
14:37:48 <pbrobinson> mtimc: the installer iso and network installer are in the same place as the pre-canned image https://download.fedoraproject.org/pub/alt/iot/
14:38:05 <pbrobinson> I have no idea what koan is
14:38:12 <mtimc> ah. ok. I'll have a look.
14:38:20 <pbrobinson> pwhalen can link to the docs for PXE setup
14:38:21 <mtimc> 'kickstart over a network'
14:38:57 <pwhalen> https://docs.fedoraproject.org/en-US/Fedora/26/html/Installation_Guide/chap-pxe-server-setup.html
14:39:38 <mtimc> pwhalen, that's pretty much what cobbler + koan did. I'll check the newer approach.
14:39:43 <mtimc> thanks
14:40:22 <pbrobinson> right, does anyone else have any issues
14:40:26 <pbrobinson> I have the following
14:40:33 <pbrobinson> #blockers for f29 iot
14:40:44 <lorbus> I have something for open floor
14:41:00 <pbrobinson> lorbus: hold, we'll get there
14:41:33 <pbrobinson> #info initial-setup should be fixed, needs testing https://bugzilla.redhat.com/show_bug.cgi?id=1646568
14:42:00 <pbrobinson> #info dracut issues on boot with no ethernet https://github.com/dracutdevs/dracut/issues/488
14:42:06 <mtimc> if f29 is based on podman 10, does that mean that we'll have to wait for f30 for podman 11?
14:42:39 <pbrobinson> #info not a blocker but discussed - issues on install/upgrade with some SD cards on RPi https://bugzilla.redhat.com/show_bug.cgi?id=1648112
14:42:59 <pbrobinson> mtimc: see point above about monthly releases
14:43:27 <pbrobinson> mtimc: unless the podman maintainers decide not to push it, I answered that in reply to your email on the list
14:44:40 <mtimc> pbrobinson, my question was more about whether major versions get changed within a release. f29 will have a 6 month gap to f30, I think, and that's what had me concerned
14:45:00 <pbrobinson> mtimc: i answered that on the email!!
14:45:19 <pbrobinson> mtimc: it depends on the maintainer's decision
14:45:42 <mtimc> let me re-read the email - I'm not sufficiently up to speed on the maintainers' process.
14:45:57 <mtimc> your meeting: ignore me.
14:46:18 <pbrobinson> #info raspberry pi3 intermittent wifi issues, mtimc to file a bug
14:46:49 <pwhalen> mtimc, in fedora its up to them, we're just consumers in the IoT edition
14:47:51 <pbrobinson> right, any other blockers
14:47:53 <pwhalen> I am not aware of anything else
14:48:18 <pbrobinson> right, aiming for a release early next week
14:48:28 <pbrobinson> #info aiming for a release early next week
14:49:00 <pbrobinson> https://bugzilla.redhat.com/show_bug.cgi?id=1648112
14:49:03 <pbrobinson> bah
14:49:12 <pbrobinson> #topic 2) ==== Open Floor ====
14:49:16 <pbrobinson> lorbus: you're up first
14:50:20 <lorbus> I'd like to put coreos-assembler on the radar for compose/build systems, in case it isn't already
14:50:34 <lorbus> https://github.com/coreos/coreos-assembler
14:51:42 <lorbus> That container is going to be used by Fedora CoreOS as multi-stage build system, on jenkins. There is a WIP PR open for building Silverblue using this tool, too
14:51:43 <pbrobinson> lorbus: how is that different to, what does it provide, on top of pungi. Does it integrate into all the Fedora release processes?
14:51:53 <lorbus> https://pagure.io/workstation-ostree-config/pull-request/109
14:52:37 <lorbus> pbrobinson: It does not, as of now. It is an alternative to the way pungi does it now
14:52:50 <pbrobinson> maybe, I'll consider it next year once it's settled down and the all the rel-eng bits are considered
14:52:56 <lorbus> I expect it to be integrated in one way or the other at some point
14:53:20 <pbrobinson> right, "at some point".... that could be years
14:53:27 <lorbus> architecturally, we are going to move away from Anaconda, and use coreos' Ignition instead
14:53:52 <pbrobinson> ATM I feel it's too early and I need time to work out the differences/pros/cons etc
14:54:04 <lorbus> sure. but as this is also the tool used to build RHCOS, there is some company commitment behind it
14:54:41 <pbrobinson> lorbus: that's fine, but frankly at the moment I don't care, next year maybe I will. ATM I need something that works *now*
14:55:03 <lorbus> I just wanted to throw that in here, as its what my team is working and I'll report back/come up with a POC when I feel its ready
14:55:24 <pwhalen> thanks lorbus, will give that a read
14:55:25 <pbrobinson> ATM ignition is missing a bunch of stuff we need now, and ignition is in my plans but not until next year
14:55:29 <lorbus> so just a FYI, no work request :D
14:56:25 <lorbus> pbrobinson: cool, think of it as the new compose & artifact build tool for ignition based system images
14:57:07 <lorbus> and yes, Ignition is heavily worked on right now
14:57:25 <mtimc> lorbus, do you have a timeline ambition for when it will be production ready?
14:58:04 <pbrobinson> lorbus: I wonder how much audit, reproducibility and compliance requirements it has built in to it.
14:58:14 <lorbus> I am not sure actually, but I think its not before Q2 2019
14:59:12 <pbrobinson> the problems we always had with the compose tools out of the ostree/atomic projects was basically they had none of that which meant it was always a no go because of things like license and audit requirements
14:59:56 <pbrobinson> right does anyone else have anything for open floor
15:00:14 <lorbus> pbrobinson: not sure about that either. I presume it will be handled, I'll do some research
15:00:20 * pwhalen has nothing else
15:00:35 <pbrobinson> lorbus: don't presume anything ;-)
15:00:55 <pbrobinson> thank you all
15:00:55 <pbrobinson> #endmeeting