15:02:27 <davdunc> #startmeeting fedora_cloud_meeting
15:02:27 <zodbot> Meeting started Thu Oct 13 15:02:27 2022 UTC.
15:02:27 <zodbot> This meeting is logged and archived in a public location.
15:02:27 <zodbot> The chair is davdunc. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
15:02:27 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:02:27 <zodbot> The meeting name has been set to 'fedora_cloud_meeting'
15:02:37 <davdunc> #topic roll call
15:02:54 <Eighth_Doctor> .hello ngompa
15:02:55 <zodbot> Eighth_Doctor: ngompa 'Neal Gompa' <ngompa13@gmail.com>
15:03:01 <davdunc> #chair ngompa
15:03:01 <zodbot> Current chairs: davdunc ngompa
15:03:27 <cjp256> .hello cjp256
15:03:30 <zodbot> cjp256: cjp256 'Chris Patterson' <cjp256@gmail.com>
15:03:35 <davdunc> hey ngompa! great to see you here.
15:03:43 <davdunc> #chair cjp256
15:03:43 <zodbot> Current chairs: cjp256 davdunc ngompa
15:04:01 <davdunc> hey cjp256 glad to see you here as well!
15:04:17 <nzwulfin> .hello mmicene
15:04:18 <zodbot> nzwulfin: mmicene 'Matt Micene' <nzwulfin@gmail.com>
15:04:33 <dduffey> .hello dduffey
15:04:34 <zodbot> dduffey: dduffey 'None' <david.duffey@treeotech.com>
15:04:45 <Eighth_Doctor> davdunc: the joys of being on PTO :)
15:04:51 <davdunc> #chair mmicene dduffey
15:04:51 <zodbot> Current chairs: cjp256 davdunc dduffey mmicene ngompa
15:05:09 <nzwulfin> w00t finally made a meeting :)
15:05:21 <davdunc> dduffey: my title officially changed this week. I now officially have your old job!
15:05:42 <davdunc> chair nzwulfin
15:05:47 <davdunc> #chair nzwulfin
15:05:47 <zodbot> Current chairs: cjp256 davdunc dduffey mmicene ngompa nzwulfin
15:06:01 <dduffey> davdunc: nice! well deserved and I bet it was easy to fill my shoes :)
15:06:28 <davdunc> thanks dduffey and no. you are a great manager.
15:06:52 <davdunc> love all these folks here for the meeting!
15:07:02 <dduffey> I should have been reporting to you :) would have saved you the headache
15:08:24 <davdunc> haha.
15:08:45 <nzwulfin> i'm aiming to be around a lot more, meetings and otherwise
15:08:53 <davdunc> #topic Action items from last meeting
15:09:25 <davdunc> davdunc track and ensure updates to the waagent rpm to include the patch for ConditionVirtualization
15:09:56 <davdunc> #info davdunc track and ensure updates to the waagent rpm to include the patch for ConditionVirtualization
15:10:09 <davdunc> okay... I didn't really track too hard. cjp256 did you?
15:10:29 <cjp256> i've got a patch ready to go, just need to open the PR
15:10:38 <davdunc> #link https://github.com/Azure/WALinuxAgent/pull/2661
15:10:43 <davdunc> isn't this the one?
15:11:23 <cjp256> yeah I updated the fedora packaging with that patch and tested it, just need to PR it
15:11:34 <davdunc> aha. so we just need the patch for the rpm.
15:12:36 <davdunc> I see. Can I action you with that task?
15:12:37 <cjp256> just having auth issues pushing it that I need to sort out :)
15:12:55 <cjp256> Sure
15:12:59 <davdunc> thanks.
15:14:13 <davdunc> #action cjp256 to add PR for the ConditionVirtualization to the RPM tree and spec updates needed.
15:14:21 <davdunc> look good?
15:14:49 <spotz> peeks in the right channel:)
15:15:10 <davdunc> #chair spotz
15:15:10 <zodbot> Current chairs: cjp256 davdunc dduffey mmicene ngompa nzwulfin spotz
15:15:12 <cjp256> perfect, thanks
15:15:14 <davdunc> Welcome!
15:15:23 <davdunc> spotz glad to have you here!
15:15:39 <davdunc> #info davdunc to come up with an outline (on the ML so we can all participate)
15:15:48 <cjp256> I also think it'd be best to upstream the fedora distro patch https://src.fedoraproject.org/rpms/WALinuxAgent/blob/rawhide/f/0001-Rudimentary-Fedora-OS-implementation.patch at some point
15:16:18 <davdunc> That sounds like a great idea.
15:16:42 * davdunc Can you add a cloud-sig ticket for that cjp256 and then we can get someone who has time on it.
15:16:58 <davdunc> sorry. wrong send key
15:17:29 <davdunc> okay.
15:17:42 <cjp256> Sure thing
15:18:10 <davdunc> for this new topic, I have some things in progress, but haven't posted to the mailing list yet.
15:18:31 <davdunc> I'll get something into the ML today so we can work on it jointly.
15:18:56 <davdunc> #action davdunc to publish outline for release blog for the cloud edition.
15:19:57 <davdunc> should read today. I'll get it out today and ready to work on for tomorrow and the weekend for anyone with time/inclination.
15:21:09 <davdunc> once that's out, mhayden can action forming what we come up with into a cohesive blog post.
15:21:09 <Eighth_Doctor> cjp256: I did see that upstream wasn't happy about "fedora-specific" stuff, so I'm not sure it's worth our time to try anymore for now
15:21:47 <Eighth_Doctor> sadly, that seems like shades of what I experienced dealing with the GCP agents long ago
15:21:50 <davdunc> Eighth_Doctor: yea. I saw that, apparently it's the testing infrastructure that they are concerned about. Anything we can do there? We should have the CI work on our radar anyway.
15:21:54 <cjp256> their concern was the patch I PR'd affected _more_ than just Fedora
15:22:08 <Eighth_Doctor> yeah, because it's technically correct
15:22:32 <Eighth_Doctor> and I am aware of plenty of people who make images with "all the agents" to let people upload to their hypervisor of choice
15:22:38 <Eighth_Doctor> usually appliance images
15:22:53 <davdunc> true. It's complicated for those governance teams because they think beyond the limits of their own infrastructure.
15:22:55 <Eighth_Doctor> heck, I literally help make one such product
15:22:55 <cjp256> Perhaps.  But Fedora is the only downstream asking to add the flag.
15:23:14 <Eighth_Doctor> cjp256: Fedora is pretty much the only distro that also strongly encourages upstream involvement
15:23:24 <davdunc> true.
15:23:25 <Eighth_Doctor> I suspect nobody else bothers
15:23:45 <Eighth_Doctor> I know that for a product I helped work on, I just monkeypatched it because I expected upstream wouldn't care
15:24:24 <davdunc> Eighth_Doctor: I know that the Debian team just basically accepts stuff for the platforms if it's for the local image only.
15:25:16 <davdunc> I think that it is a precedent that we really want to set.
15:25:35 <Eighth_Doctor> AWS is sadly both good and bad in this case
15:25:45 <Eighth_Doctor> it's great in that it doesn't use weird custom hypervisor technology
15:25:58 <Eighth_Doctor> but it's bad that it's really hard to detect to disable their agents properly when booting in a non-AWS environment
15:26:30 <davdunc> again, the agents aren't expected to be specific to the hypervisor in some cases, like the Amazon SSM agent.
15:26:43 <Eighth_Doctor> sure
15:26:53 <Eighth_Doctor> but some are
15:27:17 <davdunc> yea. there are some good examples of both in all of the providers.
15:27:23 <Eighth_Doctor> and GCP and Azure are examples of specialized agents that really are hypervisor specific
15:27:47 <davdunc> that's a key difference in this case, yes.
15:28:12 <cjp256> fwiw i just installed google-guest-agent on my laptop and it started on boot.
15:28:20 <cjp256> I personally don't think ConditionVirtualization= is necessarily the right approach in all cases, but I do believe that the agent should handle any environment gracefully.  I'd say if there are issues that's where the upstream involvement should be.
15:28:21 <Eighth_Doctor> there's no specification for how guest-host interfaces should work in cloud environments :(
15:28:43 <davdunc> yea. That's not really something for which there is a standard.
15:29:05 <davdunc> spotz: are we clobbering your meeting time right now/
15:29:09 <davdunc> ?
15:29:12 <Eighth_Doctor> it probably isn't the right approach everywhere, but the Azure agents are required for both local Hyper-V and Azure
15:29:20 <Eighth_Doctor> and so it correctly applies here
15:29:27 <Eighth_Doctor> as for the google one, sighs
15:29:35 <spotz> davdunc: I'm just rotating through all the IRC channels/servers I have meetings going on:)
15:29:57 <Eighth_Doctor> what's not helping in Microsoft's case is their track record with security issues
15:30:00 <davdunc> got it spotz !
15:30:13 <spotz> :)
15:30:15 <Eighth_Doctor> that's making FESCo much more wary about allowing it to be enabled without some kind of guard
15:30:50 <cjp256> walinuxagent is not required for local hyperv afaik
15:30:56 <davdunc> Eighth_Doctor I see that point, but that means that if we are putting patches on top of their work and we run into a CVE, we may end up with a fast change requirement due to our own specification.
15:31:16 <davdunc> cjp256: not required, but it is recommended, no?
15:31:22 <cjp256> In fact I don't think it'll be at all useful for local hyperv. and that's where ConditionVirtualization fails to match the correct granularity
15:31:48 <davdunc> that answered my question. :)
15:31:50 <Eighth_Doctor> I have been advised by Microsoft before that it's required for correct operation of VMs on Hyper-V
15:32:13 <davdunc> do we have documentation for this?
15:32:17 <cjp256> I suspect you were misinformed :D
15:32:25 <Eighth_Doctor> well, tell that to Microsoft :)
15:32:43 <themayor> hey sorry im late
15:32:54 <Eighth_Doctor> they insisted that the enlightenments and integrations to make the VM perform properly on Hyper-V only work if the agent is installed
15:32:56 <dduffey> themayor, explain yourself :)
15:33:09 <spotz> hehe
15:33:11 * dustymabe lurks
15:33:14 <Eighth_Doctor> they compared it to qemu-guest-agent for KVM
15:33:17 <Eighth_Doctor> so 🤷‍♂️
15:34:07 * davdunc themayor: local hyper-v WAAgent or no?
15:35:41 <dduffey> cjp256 maybe first step would be to have a base cloud image (not Azure/Hyper-V specific) as a VHD?
15:36:04 <davdunc> dduffey: we can do that!
15:36:23 <Eighth_Doctor> we don't need to do that
15:36:27 <davdunc> right.
15:36:30 <Eighth_Doctor> we already produce raw disk images
15:36:44 <Eighth_Doctor> unenlightened disk images can be imported into Azure or Hyper-V now
15:37:13 <Eighth_Doctor> producing VHDs is just overkill if we're not adding specific Azure content
15:37:37 <davdunc> Eighth_Doctor: it might help to look at the requirements though.
15:37:39 <Eighth_Doctor> and keep in mind, we have to produce the space-expensive VHDs for Azure, since VHDXs aren't supported there
15:37:41 <davdunc> #link https://learn.microsoft.com/en-us/azure/virtual-machines/linux/create-upload-generic
15:38:03 <davdunc> there are specific sizing requirements there it looks like.
15:38:51 <davdunc> "VHD images on Azure must have a virtual size aligned to 1 MB. Typically, VHDs created using Hyper-V are aligned correctly."
15:38:56 <Eighth_Doctor> yeah, if we produce VHDs then we'd do that
15:38:56 <cjp256> when i tested the Azure image I converted it to vhd with `qemu-img convert -o subformat=fixed,force_size -O vpc ...`
15:39:26 <davdunc> cjp256: ack.
15:39:28 <Eighth_Doctor> iirc, we should have UDF support enabled in our kernel
15:40:35 <davdunc> I don't remember, but we can look at that.
15:40:37 <cjp256> but yeah, I agree shipping a generic VHD image would be a nice option
15:40:55 <dduffey> cjp256 would user credentials get provisioned correctly?
15:40:58 <davdunc> #action davdunc to verify UDF support in the kernel
15:41:11 <cjp256> Yeah I don't recall any provisioning issues
15:41:33 <themayor> it would be nice. i think we should start with a small step to at least get things off the ground
15:41:50 <Eighth_Doctor> we currently have no way to produce VHDs right now
15:42:04 <themayor> ideally we should have the WALA agent in there as a bare minimum but if we need to talk to that team a bit more, we shouldnt let it be a blocker
15:42:05 <Eighth_Doctor> our current image creation tooling doesn't support it, afaik
15:42:33 <davdunc> Eighth_Doctor: that's correct and I am still working on the Mash support.
15:44:16 <themayor> Eighth_Doctor: can't we just run a conversion step on the base image once its produced to spit out a VHD?
15:44:17 <cjp256> here's the Azure kickstarter https://pagure.io/fedora-kickstarts/blob/main/f/fedora-cloud-base-azure.ks
15:44:32 <Eighth_Doctor> themayor: I don't know where we'd run it.
15:44:39 <Eighth_Doctor> blech
15:44:45 <davdunc> yea. koji isn't really ideal for that.
15:45:04 <Eighth_Doctor> if we did switch to kiwi, then kiwi would produce the correct artifact for us
15:45:05 <themayor> how about we spin up something on azure that can detect a new image and run the conversion?
15:45:30 <themayor> yes but switching to kiwi is a larger discussion and project now ;)
15:45:42 <themayor> not that i agree/disagree
15:45:54 <davdunc> Eighth_Doctor: kiwi is integrated in koji now, right?
15:46:07 <Eighth_Doctor> mostly
15:46:13 <Eighth_Doctor> still some bugs to work out and being tested in CentOS
15:46:23 <Eighth_Doctor> but the integration is there
15:46:54 <davdunc> Yes. I am aware of the testing, but haven't been following aarfab's results. spotz might have more detail there.
15:47:15 <spotz> I can ask but don't have one
15:47:32 <davdunc> spotz: all good. didn't expect a whole lot more. I can ask too. :D
15:47:40 <spotz> :)
15:48:10 <Eighth_Doctor> this is the current status: https://pagure.io/centos-infra/issue/696#comment-821158
15:48:32 <davdunc> themayor: it's not much of a larger discussion. We can't use the osbuild stuff yet because they don't have plans to support the btrfs work we have done.
15:49:01 <davdunc> #link https://pagure.io/centos-infra/issue/696#comment-821158
15:49:35 <Eighth_Doctor> having kiwi will also allow us to produce WSL images eventually
15:49:39 <davdunc> okay Eighth_Doctor it looks like "miles to go before we sleep."
15:49:52 <Eighth_Doctor> we will be able to produce any kind of artifact except ostrees
15:50:07 <Eighth_Doctor> and that's probably fine, since rpm-ostree uses it's own world of tooling anyway
15:50:09 <davdunc> #chair dustymabe
15:50:09 <zodbot> Current chairs: cjp256 davdunc dduffey dustymabe mmicene ngompa nzwulfin spotz
15:50:12 <Eighth_Doctor> s/it's/its/
15:50:17 <davdunc> sorry dustymabe I forgot to get you in there.
15:51:07 <davdunc> so are we looking at building the Azure image for right now with cloud-init instead of the waagent?
15:51:39 <dduffey> I will give my proxy to themayer and cjp256
15:51:48 <davdunc> got it.
15:52:41 <dduffey> I don't see any hard dependencies on waagent at the moment
15:52:55 <Eighth_Doctor> waagent and cloud-init are complementary, afaik?
15:53:03 <dduffey> yes
15:53:09 <davdunc> we are getting close to time and we haven't covered much else. I want to make sure we get some resolution here for the short term. I'd like to have an image published for 36 at the very least.
15:53:22 <davdunc> Eighth_Doctor: they are definitely so.
15:53:26 <dduffey> and some functionality will be missing (such as user admin through azure portal)
15:53:48 * davdunc yea. that's something we can include with modification after boot, isn't it?
15:55:15 <davdunc> if we can update the instances to include the management actions, I'm happy to move to using cloud-init for the initial images and then publishing images with the waagent once it is available with a local change.
15:55:20 <dduffey> cjp256 and themayor I am okay with base image that is VHD, doesn't have to be labeled Azure specific, it's a stepping stone
15:55:31 <themayor> exactly
15:55:34 <themayor> agreed
15:56:37 <davdunc> #proposed Use cloud-init for the images initially published to Azure
15:56:49 <davdunc> vote?
15:57:20 <davdunc> +1 for me obviously.
15:57:22 <dduffey> you have three folks from Microsoft here, so will put to themayor
15:57:34 * Eighth_Doctor shrugs
15:57:38 <Eighth_Doctor> it's a start
15:57:38 <Eighth_Doctor> +1
15:57:43 <themayor> dduffey: :)
15:57:47 <davdunc> :) you all count as contributor to me.
15:58:02 <dduffey> +1
15:58:14 <themayor> +1
15:58:33 <davdunc> anyone have any objections that they want to voice?
15:58:47 <davdunc> I heard yours Eighth_Doctor
15:59:07 <cjp256> +1
15:59:30 <davdunc> invoking sgallagh
15:59:31 <Eighth_Doctor> I think people will be disappointed by the experience if they except Azure goodness
15:59:36 <Eighth_Doctor> *expect
15:59:50 <Eighth_Doctor> but we can do it anyway if we can resolve the waagent thing quickly enough
15:59:59 <themayor> start small. gather feedback. improve.
15:59:59 <Eighth_Doctor> besides, if we ship that patch downstream, fesco will let us have the preset
16:00:15 <Eighth_Doctor> and we can go forth and ship it altogether anyway
16:00:18 <themayor> im all for that too, but assuming it doesnt end up holding things up
16:00:31 <Eighth_Doctor> while I would prefer to have it upstreamed, I don't care about blocking on upstreaming
16:00:54 <Eighth_Doctor> my experience with public cloud providers is that generally these kinds of things will be hard to get fixed, so we should be prepared to maintain integration patches downstream
16:00:58 <davdunc> yea. we can make a local change for the F38 release to include it and if we need to fall back to cloud-init again, that's not the worst thing in the worl.d.
16:01:16 <davdunc> world*
16:01:19 <Eighth_Doctor> we should get the Change proposal out for F38 to offer Azure images
16:01:19 <cjp256> we can go back to just enabling waagent for strictly the Azure variant ;D
16:01:40 <Eighth_Doctor> that will be the first release with the preset for waagent, and we can ship it in the Azure image
16:02:01 <davdunc> oyep.
16:02:07 <davdunc> okay. we are over time.
16:02:31 <davdunc> #agreed We will ship a first image for Azure with cloud-init enabled.
16:03:09 <davdunc> okay. I am going to go ahead and end the meeting so the packaging meeting can happen.
16:03:17 <davdunc> Thanks everyone for being here.
16:03:30 <davdunc> #info we are GREEN for cloud edition.
16:03:30 <cjp256> \o have a good day!
16:03:37 <davdunc> #endmeeting