16:30:38 <dustymabe> #startmeeting fedora_coreos_meeting
16:30:39 <zodbot> Meeting started Wed Oct 26 16:30:38 2022 UTC.
16:30:39 <zodbot> This meeting is logged and archived in a public location.
16:30:39 <zodbot> The chair is dustymabe. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:30:39 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:30:39 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:30:44 <dustymabe> #topic roll call
16:30:46 <dustymabe> .hi
16:30:47 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:31:18 <fifofonix> .hi
16:31:20 <lucab> .hi
16:31:20 <zodbot> fifofonix: fifofonix 'Fifo Phonics' <fifofonix@gmail.com>
16:31:23 <zodbot> lucab: lucab 'Luca BRUNO' <lucab@redhat.com>
16:31:56 <dustymabe> #chair fifofonix lucab jlebon
16:31:56 <zodbot> Current chairs: dustymabe fifofonix jlebon lucab
16:31:57 <jlebon> .hello2
16:31:58 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:32:07 <adamw> .hello adamwill
16:32:08 <zodbot> adamw: adamwill 'Adam Williamson' <awilliam@redhat.com>
16:32:10 <mnguyen> .hello mnguyen
16:32:11 <zodbot> mnguyen: mnguyen 'Michael Nguyen' <mnguyen@redhat.com>
16:32:32 <travier> .hello siosm
16:32:33 <zodbot> travier: siosm 'Timothée Ravier' <travier@redhat.com>
16:33:08 <dustymabe> #hcair adamw mnguyen travier
16:33:11 <dustymabe> sigh
16:33:14 <travier> :)
16:33:18 <dustymabe> #chair adamw mnguyen travier
16:33:18 <zodbot> Current chairs: adamw dustymabe fifofonix jlebon lucab mnguyen travier
16:34:02 <dustymabe> ok let's get started here shortly
16:35:07 <gursewak> .hi
16:35:08 <zodbot> gursewak: gursewak 'Gursewak Singh' <gurssing@redhat.com>
16:35:12 <jbrooks> .hello jasonbrooks
16:35:13 <zodbot> jbrooks: jasonbrooks 'Jason Brooks' <jbrooks@redhat.com>
16:35:16 <dustymabe> #chair jbrooks gursewak
16:35:16 <zodbot> Current chairs: adamw dustymabe fifofonix gursewak jbrooks jlebon lucab mnguyen travier
16:35:19 <dustymabe> #topic Action items from last meeting
16:35:27 <dustymabe> We had one action item:
16:35:33 <dustymabe> * bgilbert will follow up on https://github.com/coreos/fedora-coreos-tracker/issues/567 re. VMware
16:35:51 * dustymabe doesn't see bgilbert around today, i'll re-action
16:36:05 <dustymabe> #action bgilbert will follow up on https://github.com/coreos/fedora-coreos-tracker/issues/567 re. VMware
16:36:20 <dustymabe> next topic:
16:36:27 <dustymabe> #topic 20221101: CRITICAL OpenSSL vulnerability
16:36:41 <bgilbert> .hi
16:36:42 <zodbot> bgilbert: bgilbert 'Benjamin Gilbert' <bgilbert@backtick.net>
16:36:46 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/1329
16:36:49 <dustymabe> welcome bgilbert
16:36:54 <dustymabe> #chair bgilbert
16:36:54 <zodbot> Current chairs: adamw bgilbert dustymabe fifofonix gursewak jbrooks jlebon lucab mnguyen travier
16:37:49 <dustymabe> ICYMI the OpenSSL team announced the discovery of a CRITICAL security vulnerability. The fix will be released next Tuesday
16:38:37 <dustymabe> which happens to coincide with when we would be switching our `testing` stream to F37 content (assuming F37 GA decision is "Go" on Thursday)
16:39:11 <travier> It's bad timing indeed
16:39:23 <dustymabe> the open question is how we want to deal with these two competing events for Fedora CoreOS
16:39:34 <travier> Normally I would have said that we should make sure to get a testing release with the fix in it the day it's released but that time it does not work
16:40:06 <dustymabe> we take security seriously, so obviously we want to add appropriate priority to the openssl bug (of which we don't know the exact details yet)
16:40:06 <travier> It kind of depends on what Fedora does
16:40:39 <bgilbert> we have additional constraints that the rest of Fedora doesn't, which are that if we ship a release on a stream, we force users to upgrade to it
16:40:42 <travier> s/does/decides for the release date
16:40:55 <travier> yep
16:40:57 <dustymabe> Regardless of what we decide here there are two things that will be a constant next week: `next` will continue to be on F37 content and `stable will continue to be on F36 content.
16:41:11 <bgilbert> so Fedora as a whole can decide to ship without the fix, and the only question is whether it affects Anaconda
16:41:27 <dustymabe> bgilbert: assuming they have automated updates enabled, yes (just wanted to qualify the "force" wording)
16:41:28 <adamw> anaconda or anything anyone might do on a live desktop.
16:41:34 <bgilbert> adamw: +1
16:42:02 <bgilbert> but for us, I think it's bad form to force users to update through a rebase with possibly unknown bugs, in order to get a critical security update
16:42:22 <bgilbert> so I think we need to consider slipping our rebase
16:42:31 <travier> critical untested security do indeed usually come with other bugs
16:42:35 <lucab> it looks like in any case we'd have to respin all of stable+testing+next
16:42:46 <travier> security releases*
16:42:48 <dustymabe> bgilbert: to counter that point: stable will continue to be on 36 so they could switch streams
16:43:03 <bgilbert> of course, it's possible that the bug doesn't affect us, e.g. if it only affects TLS servers
16:43:12 <adamw> ah, i see where you're coming from. the idea is you want a `testing` build that's 36 with the security fix, so people can use that if they don't want to go to 37 yet?
16:43:15 <bgilbert> dustymabe: correct.  but we do try to keep all streams usable and secure
16:43:20 <bgilbert> adamw: right
16:43:26 <dustymabe> bgilbert: indeed
16:43:38 <bgilbert> that could be as simple as: ship the fix, then _immediately_ ship the 37 rebase
16:43:59 <dustymabe> this is one option ^^
16:44:13 <aaradhak> .hi
16:44:14 <zodbot> aaradhak: aaradhak 'Aashish Radhakrishnan' <aaradhak@redhat.com>
16:44:33 <Nemric> hi
16:44:34 <bgilbert> lucab: and depending on the size of the fix, we may want it to bake a bit before shipping to `stable`
16:44:43 <dustymabe> #chair aaradhak Nemric
16:44:43 <zodbot> Current chairs: Nemric aaradhak adamw bgilbert dustymabe fifofonix gursewak jbrooks jlebon lucab mnguyen travier
16:44:58 <travier> yes, so that week after we can have 36 + openssl fixed in stable and the 37 in testing with the fix again?
16:45:16 <adamw> i'm assuming nobody is sitting on back channel info about the nature of the vuln? i know i'm not, unfortunately
16:45:38 <lucab> we can also do a very short rollout on testing, i.e. a few hours. So that we get both better signal for stable and a cleared path for the F37 rebase.
16:45:52 <dustymabe> adamw: I think we're on the same playing field, but I will point out that even if I did know I couldn't tell you :)
16:46:12 <adamw> officially, anyway
16:46:23 <travier> or I would have to kill you :)
16:46:33 <dustymabe> .fire adamw
16:46:34 <zodbot> adamw fires adamw
16:46:37 <bgilbert> hah
16:46:42 <travier> :D
16:46:45 <dustymabe> #recursion
16:47:21 <jlebon> haha
16:47:29 <bgilbert> adamw: I am not
16:47:35 <travier> anyway, if it's something that breaks TLS security for ignition then that's bad
16:47:49 <bgilbert> travier: Go crypto/tls isn't OpenSSL
16:47:50 <dustymabe> ok - let's clear one hurdle - does anyone think we should ship the 37 rebase on tuesday (before the announcement of the ssl fix and also assuming F37 is "Go")
16:48:13 <travier> isn't go using the openssl backend in Fedora. Maybe only in RHEL?
16:48:14 <bgilbert> dustymabe: let's qualify that
16:48:22 <travier> or only for fips?
16:48:29 <bgilbert> travier: wait, there's an openssl backend?
16:48:48 <travier> yes, the only wait to get FIPS in go AFAIK :)
16:48:53 * dustymabe is glad to have experts here to guide this discussion
16:48:53 <bgilbert> ahh
16:49:15 <bgilbert> does anyone think we should ship the 37 rebase before we know what's in the OpenSSL announcement?
16:49:25 <travier> https://developers.redhat.com/blog/2019/06/24/go-and-fips-140-2-on-red-hat-enterprise-linux
16:49:40 <travier> but well, we don't use FIPS in fcos so we likely don't care then
16:50:08 <travier> https://developers.redhat.com/articles/2022/05/31/your-go-application-fips-compliant
16:50:11 <bgilbert> ...ah
16:50:28 <travier> it's crazy but it's fips anyway so...
16:50:44 <bgilbert> right.  yeah, we don't support FIPS in FCOS: https://github.com/coreos/fedora-coreos-tracker/issues/302
16:51:54 <travier> https://developers.redhat.com/articles/2022/05/31/your-go-application-fips-compliant > this one seems to confirm that this is not the case on Fedora so we should be good for ignition
16:52:02 <dustymabe> ok - there is more to discuss on this topic but let's clear one decision so we can further the discussion
16:52:06 <travier> what else do we have in the boot logic using openssl?
16:52:39 <travier> if we have nothing directly at risk, then we can "just" follow the Fedora release schedule
16:52:40 <bgilbert> travier: I don't think boot vs. runtime really matters?  FCOS doesn't use separate bootimages like RHCOS does
16:52:45 <dustymabe> we do support grabbing the `rootfs` from a remote
16:52:52 <bgilbert> +1 dustymabe
16:53:10 <travier> boot logic was a shortcut indeed
16:53:41 <bgilbert> but yeah, most of our network code is Go or Rust
16:53:49 <dustymabe> I think trying to thread the needle of if we want to ship this or not might cause us more grief than it's worth. I'm just assuming we're going to want to ship it. However I do think it's enlightening to go through the exercise of thinking about what uses OpenSSL.
16:54:18 <travier> we have some curl commands somewhere in dracut scripts maybe?
16:54:19 <lucab> I'd prefer shipping the openssl update first (whatever that is) and then doing the F37 rebase for testing
16:54:22 <bgilbert> interestingly, the rootfs fetch is written to not have to care about TLS integrity
16:54:23 <dustymabe> I'd hate to do the analysis and then be wrong about it
16:54:30 <dustymabe> bgilbert: good point
16:54:34 <bgilbert> (though RCE is obviously still relevant)
16:54:51 <dustymabe> anyway I'm going to throw out a proposal
16:54:52 <dustymabe> #proposed assuming F37 GA is Nov 1st we will hold the `testing` rebase to F37 content one week while we ship security updates for the recently announced OpenSSL CVE.
16:55:03 <travier> how would shipping the update + then the rebase work?
16:55:19 <bgilbert> users might potentially have a systemd service that curls something, so I think we have to prioritize any client-side fixes
16:55:21 <dustymabe> travier: ship the update next week, ship the rebase the following week (for `testing`)
16:55:22 <travier> lucab: ^^
16:55:51 <travier> so we delay if fedora does not delay?
16:56:02 <lucab> travier: what part specifically? the timing of the rebase?
16:56:12 <bgilbert> gaming this out: worst case, there's a large code fix, we're affected, we decide to slip.  we build next/testing as soon as the fix hits bodhi, 24 hour rollout, then we ship a stable update.  throw in 24 hours for pipeline problems and we've used up the whole week.
16:56:17 <travier> yes, as dusty said I assume?
16:56:18 <dustymabe> travier: correct - as bgilbert mentioned we are in a bit more control of our users updates, so I think it can make sense to differ here
16:56:27 <bgilbert> intermediate case, we can ship to all channels at once.  best case, we don't even have to care.
16:56:31 <bgilbert> s/channels/streams/
16:56:53 <jlebon> bgilbert: +1
16:57:04 <travier> in general I agree that we should delay our rebase to 37 if the issue is significant enough
16:57:10 <dustymabe> the current proposal is for `testing` stream only - obviously once we make this decision we can talk about the other streams
16:58:03 <fifofonix> +1 for proposal.  end users using ssh will use openssl right?
16:58:15 <jlebon> bgilbert: "we don't even have to care" -> can you clarify that?
16:58:43 <travier> fifofonix: not sure how much of openssl is used in openssh
16:59:03 <bgilbert> jlebon: the example I gave was that the bug is in TLS server-side code.  arguably we're unaffected in that case
16:59:16 <lucab> as I said I think we will be forced to make updates on all streams with the openssl fix quite quickly, pretty much every user will be asking about that
16:59:16 <dustymabe> I think ssh has its own crypto, but I think it does support auth via certificates (but I'm definitely no expert here)
16:59:22 <bgilbert> fifofonix: yeah, depends where the problem is.  crypto primitives yes, TLS no AFAIK
16:59:37 <fifofonix> fifofonix only uses certificates
17:00:04 <bgilbert> because of the potential for timing variations in a rushed three-channel staged release, I'm thinking if we decide we need to slip, we just slip by a full week
17:00:06 <jlebon> bgilbert: i meant schedule-wise. if it's best case, we ship it in next (because why not) and otherwise don't do an "async" release for testing/stable?
17:00:58 <bgilbert> but arguably we don't need to decide in advance that we _will_ slip the release.  we can wait to see what the problem is
17:01:09 <dustymabe> the reason I'd like to assume the worst case here is because it allows us to take control of our schedule back, rather than relying on unknown information
17:01:23 <bgilbert> jlebon: we'd normally have triple builds already lined up for the Tuesday release, right?  so we wouldn't ship in any streams
17:01:29 <bgilbert> on Tuesday
17:01:43 <bgilbert> *Tuesday rebase
17:01:53 <jlebon> bgilbert: ahh gotcha. so don't delay the rebase and don't fast-track it anywhere.
17:01:53 <bgilbert> dustymabe: true
17:02:04 <bgilbert> jlebon: we have that option, yeah
17:02:12 <travier> So, our stance would be to delay the rebase to 37 unless it's a minor enough issue
17:02:34 <dustymabe> travier: i'd prefer to not put off the decision on ^%^
17:03:01 <travier> ahh gotcha. so don't delay the rebase and don't fast-track it anywhere. > ?
17:03:11 <dustymabe> we'll need to communicate this (change in rebase date) to our users anyway and I'd rather not be confusing and say "unless it's not bad"
17:03:34 <bgilbert> I don't feel 100% great about putting off a major scheduled release solely because some project made spooky Halloween noises
17:03:48 <bgilbert> but point taken about scheduling certainty
17:04:00 <jlebon> dustymabe: given that disclosure is tuesday, there's still room in the week to adapt,
17:04:01 <dustymabe> at least the release date isn't April 1st
17:04:31 <dustymabe> jlebon: yes, we *can* adapt, but do we want to?
17:04:31 <lucab> by the way https://twitter.com/mattdm/status/1585285318265262081
17:04:32 <jlebon> do we know if it's beginning of day or EOD?
17:04:56 <bgilbert> upstream announcement is due by 1 PM ET
17:05:04 <jlebon> dustymabe: i'm not saying we rush to get things out, but it should be enough time to do the rebase as we would've
17:05:25 <dustymabe> jlebon: there's more that goes into it than I'd like to sign up for
17:05:40 <bgilbert> dustymabe: how so?
17:06:18 <bgilbert> we could have builds for the triple ready to go, then have a go/no discussion, and possibly abandon them
17:06:22 <dustymabe> bgilbert: one example.. assuming we ship a newer f36 then we'd get newer packages in `next` too
17:06:25 <bgilbert> there'll be some delay while we're waiting for fixed packages anyway
17:06:56 <bgilbert> dustymabe: and?
17:07:17 <dustymabe> it's just a whole new cycle of possible test failures with new packages and investigations
17:07:41 <dustymabe> right now F37 content has been frozen for weeks
17:07:48 <bgilbert> dustymabe: but isn't that true whether or not we reserve the option to not slip?
17:07:50 <dustymabe> all of that new content is going to come in as soon as GA ships
17:07:59 <dustymabe> no
17:08:11 <dustymabe> if we don't slip then we ship mostly GA content on Tuesday
17:08:14 <bgilbert> lucab: +1, maybe the whole discussion ends up being irrelevant
17:08:29 <jlebon> dustymabe: we could hold content bumps until the rebase clears
17:08:29 <dustymabe> if we do slip I think we'd need to ship latest f37 content (all 0-day updates included)
17:09:06 <dustymabe> I think there are a lot of things we *can* do (our tooling is amazing). I think we should weigh cost vs benefit here
17:09:13 <bgilbert> strictly speaking we could choose not to promote `next` and treat this one as an OOB release, but leaving that aside
17:09:37 <dustymabe> bgilbert: indeed, that is an option. The content set gets so out of date at that point, though
17:09:42 <bgilbert> yeah
17:09:56 <dustymabe> package set has already been frozen multiple weeks as of today
17:10:05 <bgilbert> oh, is your point that if we know we're going to slip, we could pull 0-day updates into next-devel in advance?
17:10:48 <dustymabe> no, my point is that if we slip we will need to ship all the 0-day updates too, so let's give ourselves a week and not stress about juggling plates next week
17:10:58 <bgilbert> but that's true anyway
17:11:18 <bgilbert> if we slip, we will need to deal with 0-day updates
17:11:26 <bgilbert> if we do not slip, we're basically already ready to go
17:11:37 <dustymabe> right
17:11:37 <bgilbert> reserving the option of not slipping doesn't require much extra work
17:11:57 <dustymabe> except the proposed triple that we would throw away
17:12:03 <bgilbert> correct
17:12:08 <bgilbert> just some builds and some f-c-c reverts
17:12:18 <bgilbert> well, one
17:13:01 <dustymabe> i think i'm mostly saying that doing a major rebase is complicated enough already.. Let's not try to throw conditional logic ino it
17:13:23 <dustymabe> we have update barriers to consider and the promotion comes directly from `next` (not `testing-devel`) etc'
17:13:37 <jlebon> given lucab's link above, i'd suggest we pick this up again in #fedora-coreos and the tracker tomorrow after go/no go
17:13:57 <dustymabe> indeed - having that information would be useful
17:14:04 <travier> +1
17:14:06 <bgilbert> yeah, and I'm trying to trade that off against allowing OpenSSL to drive our narrative around Fedora rebases
17:14:15 <travier> let's make an ad-hoc meeting tomorrow?
17:14:17 <bgilbert> especially since this only matters in the case that Fedora decides not to allow that
17:14:23 <bgilbert> +1
17:15:02 <dustymabe> sounds good
17:15:21 <bgilbert> when's the Fedora Go/No Go?
17:15:28 <dustymabe> it will be late in the day that the meeting ends
17:15:35 <dustymabe> i think it starts around this time and can go for hours
17:15:42 <travier> I don't like it as well but I also don't like bundling a vuln fix with a significant rebase
17:15:43 <bgilbert> can't imagine why :-P
17:15:54 <dustymabe> hah
17:16:04 <travier> we already had this issues with kernel vuln and version rebases
17:16:12 <bgilbert> travier: yeah
17:16:41 <travier> so I'd prefer we slip one week to comfortably ship the fix and let the rebase happen after
17:16:45 <dustymabe> travier: hmm. I don't remember this exact problem coming up before. Usually once we find out about a vuln the fix is already available
17:17:13 <bgilbert> dustymabe: I think he meant having to backport a fix to a previous kernel stable series
17:17:15 <travier> one time we had a kernel vuln, but the fix was only in the package for the next kernel version
17:17:41 <travier> and there was a regression in the new kernel version
17:17:42 <dustymabe> #info we will meet again tomorrow once Fedora decides if F37 will ship next week to discuss the scenarios here.
17:17:52 <travier> thus there was no option for users hit be the regression
17:18:36 <dustymabe> travier: I think I see your point
17:18:41 <dustymabe> took me a moment
17:18:54 <dustymabe> move on to next topic?
17:18:55 <travier> if there is a regression in openssl and in F37 and we ship it at the same time as the 37 rebase, then there is no option for users to stay on F36
17:19:23 <travier> (with a fixed openssl)
17:19:29 <dustymabe> travier: not exactly.. users will have the option of `stable` (F36 content) and/or rollback
17:19:43 <travier> but without the openssl fix
17:20:04 <bgilbert> or layering.  but those aren't usually in scope as preferred solutions
17:20:08 <dustymabe> travier: I imagine we'll be shipping the fix to stable soon enough as well, but yes, after some trivial amount of time in `testing`
17:20:40 <travier> I think we can decide already that we'll slip by a week
17:20:41 <dustymabe> bgilbert: agree.. at least users do have options though
17:20:49 <travier> minimum
17:21:01 <bgilbert> thanks for the discussion, all.  these are the decisions that make us a distro rather than a pile of code <3
17:21:23 <dustymabe> travier: I think I agree, but we'll hash it out more tomorrow when we have the "Go/No Go" information
17:21:33 <travier> but let's see tomorrow. I think we should bring our discussion to the meeting tomorrow
17:21:34 <dustymabe> #topic including audit in Fedora CoreOS
17:21:40 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/461
17:21:48 <travier> We are becoming an edition with F37
17:21:54 <travier> our opinion will matter in this meeting
17:22:05 <dustymabe> not sure if we have enough time to discuss this $topic properly
17:22:17 <travier> let's move it to next week
17:22:23 <dustymabe> +1
17:22:27 <dustymabe> #topic open floor
17:22:28 <travier> I'll remake a new ticket with the info and status
17:22:50 <dustymabe> anyone with anything for open floor?
17:23:13 <bgilbert> travier: hmm, that's an interesting angle.  I'm not sure about blocking other editions because of our additional constraints though
17:23:26 <dustymabe> should we coordinate when to meet tomorrow? - should we just say to watch the other meeting (happens in IRC) and then convene in #fedora-coreos after decision has been made?
17:23:40 <travier> not saying we'll block editions, just saying that we'll share our perspective
17:23:45 <travier> other*
17:24:11 <dustymabe> i'm +1 to sharing perspective, but also educating people on the ability of our tooling and the power of our streams model
17:24:19 <bgilbert> dustymabe: once the decision comes down, let's give 30 or 60 minutes notice in #fedora-coreos before discussing?
17:24:35 <bgilbert> travier: yeah, indeed
17:24:51 <dustymabe> to let them know that it's not a big deal if they do decide to ship, our users still have options with `next` F37 and `stable` F36
17:25:02 <bgilbert> dustymabe: disagree
17:25:13 <dustymabe> bgilbert: +1 to 30 to 60 minute window (will try to make ti start on the half or turn of an hour
17:25:35 <bgilbert> let's not use the educational opportunity to be misleading about how to engage with the FCOS update model
17:26:03 <bgilbert> i.e., it's better to say that FCOS will slip if necessary than to say "here are the things you can do to work around our decisions"
17:26:58 <dustymabe> i.e. letting people know that "if you want f37 you can find it over here on the `next` stream and here are the implications of that" isn't something we should do?
17:27:11 <bgilbert> oh, that's what you mean by options
17:27:19 <travier> I'm approaching that from a different perspective: we'll share the risks we have to consider and then discussion we've had
17:27:29 <bgilbert> dustymabe: sorry, yeah, +1
17:27:46 <dustymabe> yeah, I mean at the end of the day if someone is unhappy that we'd slip a week we can just tell them you can already get what you want
17:27:51 <bgilbert> right
17:27:55 <bgilbert> travier: wfm
17:28:15 <dustymabe> As Ian McLeod would say: "I think we're all in violent agreement here".
17:28:30 <travier> I'm trying to find the time it's scheduled but I can not so far
17:28:36 <travier> :)
17:28:55 <dustymabe> Ben Cotton: usually puts out an email
17:29:04 <bgilbert> #link https://calendar.fedoraproject.org/Fedora%20release/#m10352
17:29:15 <bgilbert> #undo
17:29:15 <zodbot> Removing item from minutes: <MeetBot.items.Link object at 0x7ffba4fe00f0>
17:29:24 <dustymabe> last week it was at 1700 UTC in #fedora-meeting.
17:29:29 <bgilbert> #link https://calendar.fedoraproject.org/Fedora release/#m10352
17:29:33 <bgilbert> #undo
17:29:33 <zodbot> Removing item from minutes: <MeetBot.items.Link object at 0x7ffba502a908>
17:29:40 <bgilbert> argh.  you'll have to paste it manually
17:29:47 <dustymabe> I assume it will be the same time this week
17:30:21 <dustymabe> bgilbert: that first link works for me
17:30:36 <bgilbert> weird, I'm getting a server-side error on the %20
17:31:11 <dustymabe> anything else for open floor ?
17:32:19 <dustymabe> #endmeeting