Episode Transcript
Transcripts are displayed as originally observed. Some content, including advertisements may have changed.
Use Ctrl + F to search
0:00
Once again today we're joined by Brent.
0:02
Hello Brent, how are you? Hey Alex, I'm good.
0:04
How are you guys? Well we're fantastic now
0:06
that you're here. It's always nice to have a little maple syrup flair. And
0:09
you know what? I just got back from El
0:11
Salvador and there was no maple syrup
0:13
to be had. Except for at Wendy's.
0:15
But you don't want their maple syrup. Makes sense. You
0:17
made it back in one piece, huh? I did. I did.
0:20
I'm feeling... Towards the end of the trip I was actually feeling pretty
0:22
good. I didn't really want to go. It's a
0:24
good school figure, right? But that's
0:27
just sort of how it goes sometimes.
0:29
But I'm glad to be back because we've got a packed
0:32
episode. Not only do we have a special guest
0:34
coming up, the core developer behind Image. But
0:37
Alex, you got some new gear
0:39
in the Homelab. It's been a veritable
0:42
circus of UPS drivers the last couple
0:44
of weeks. Not only did I get a new 3D
0:46
printer, I got one of the Bamboo Lab ones. Which
0:48
is awesome, by the way. I
0:51
also got my 45 Homelab
0:53
server review unit sat on the desk here
0:56
behind me.
0:58
Big and chunky boy on the desk.
1:00
It's like a 4U box. I'm
1:03
trying to
1:04
describe to you through the
1:06
medium of audio just how big this thing is. It's
1:09
like one of those fractal, defined
1:11
size cases. Just flipped on its side and designed
1:14
to go in a rack. They do ship
1:17
rubber feet to go with this thing if you want to stand
1:19
it upright vertically. One
1:21
of the things I really like about it is on the front
1:23
there's a pair of captive thumbscrews. To
1:26
get inside, they're on the front
1:28
of the case. Imagine it's in a rack situation.
1:31
You've got two captive thumbscrews that you twiddle.
1:35
Once they're loose, you just push the top of the case
1:37
back a little bit and pop it up. Then you've got access
1:39
to all of the internals of the case. They
1:41
sent me the fully loaded unit. The
1:43
one with the 6-core Xeon CPU in it, 32GB of RAM, 1TB NVMe
1:45
SSD.
1:50
Also, the 15 pre-wired
1:53
drive bays, which is just fantastic.
1:56
In this case, the build quality is the
1:58
first thing that struck me.
1:59
It is absolutely flawless.
2:02
This thing is built like a tank, really
2:04
thick. I assume steel, metal
2:07
of some description, I assume it's steel. Really
2:09
thick, good quality machining, no
2:11
tool marks. The powder coat job is fabulous.
2:15
It's up there with System 76's
2:17
build quality from when we took a tour of their factory
2:20
a few years ago. I don't know about you Chris, but I'm feeling
2:22
all sorts of envy. I know, I'm thinking,
2:24
boy, this sounds like a pretty nice rig.
2:26
I actually do think I'd wanna have it standing out
2:29
on the table or something. Maybe I wouldn't want it in the rack. Within
2:32
arm's reach, like Alex, you just give it a slap to
2:34
prove it's actually there. Yeah, so you can give it a good smack, yeah.
2:37
So that's the aesthetics of it. Have you taken
2:39
it much further than that? Just first impressions I assume
2:41
so far. Well, the CPU that's in there is
2:43
a Xeon 3204 from 2018 with
2:47
a 1.8 or 1.9, the internet can't quite agree on
2:52
the core base clock of this CPU. 32 gigs
2:54
of RAM that's in there is ECC memory
2:57
and the one-tier SSD is an
2:59
NVMe one, as I mentioned. I
3:01
find it really interesting that they went with a Xeon CPU
3:04
and there's obviously some reasons why you would
3:06
do that. If you look at the PCIe lanes
3:08
that Xeon affords you, it basically
3:11
enables them to put the HBA
3:13
card directly onto the Super Micro motherboard
3:16
that ships in this thing. It's an
3:18
X11 SBH motherboard. I opted for the one
3:20
with the SFP plus 10 gig pair
3:22
of NICs in it. They do make a
3:25
standard RJ45 variant
3:27
as well. Alex, I heard you say 2018 in there. Does
3:31
that seem quite old these days? Yes,
3:35
and there is no way around that. It's a six-core
3:37
CPU with no media
3:39
encoding engines either. So there's no quick sync with
3:41
this chip. That single
3:43
decision led me down a bit of a path
3:46
of thinking who is this box for? Because
3:49
when we had Doug and Mitch on a couple of weeks ago, talking
3:52
about how it's big, strong, fast, stable
3:54
storage, all that kind of stuff, the
3:56
performance for just storage
3:59
is really... But
4:02
the minute I have a server on in my house
4:04
24-7, I'm looking at what else can
4:06
it be doing. I want to be doing image,
4:08
I want to be running image on it and doing a bunch of machine learning
4:11
tasks on it. I want to be doing Jellyfin
4:13
or Plex or whatever and doing a bunch of transcoding on it
4:15
maybe for remote
4:18
media streaming, that kind of stuff. And
4:20
I find the choice of a 6-core
4:22
CPU with a sub 2GHz clock
4:25
speed from 5 years ago peculiar.
4:29
I understand that the choice of motherboard
4:31
is probably what precipitated the choice of
4:33
CPU because
4:36
certain motherboards have certain features like HBA's
4:38
built into them like this one does and it supports
4:40
up to 2TB of RAM because it's a Xeon
4:43
chip. All this kind of stuff.
4:45
There's a lot of very valid reasons for
4:47
doing a Xeon. However,
4:51
if you're putting this in a home lab, it's in
4:54
the name. It's probably going in my home.
4:57
And for most people at home, you
4:59
are using a box with 15 hard
5:01
drive slots to store some kind of media,
5:04
probably. You might be a video
5:06
editor. I can't imagine that's a huge
5:09
use case, although certainly there will be
5:11
a bunch of YouTubers that use this as
5:13
their primary video editing
5:15
storage system. But I've got to imagine
5:17
that most people building these or using these
5:20
cases to build servers
5:22
are doing things like Unraid and TrueNAS
5:25
and Perfect Media Service, dare I say
5:27
it, all that kind of stuff. And so why wouldn't
5:29
you put a CPU in it that
5:32
has hardware media encoding?
5:35
It really confuses me as
5:37
to the identity of the product
5:39
a little bit. I bet it is predicated
5:42
on the motherboard. And I wonder if
5:44
just describing which motherboard and CPU
5:47
are coming in the system sort of solves this problem so then
5:49
you just know before you pull the trigger here. And
5:52
maybe some folks don't care about QuickSync. I
5:54
mean that is a thing, right? You can just buy the
5:57
chassis on its own, the case on its own, for...
6:00
about $800. So the fully loaded unit that
6:02
they sent me, which full disclosure I did
6:04
not pay for, the full unit
6:07
is $2,000. They sell a version of just the case
6:09
for about 900 and then the
6:12
case and the power supply for 900 and then
6:14
just the case is about 800. Right.
6:17
So you could, I guess, get that one and put your own motherboard
6:20
and CPU in. Absolutely, yeah. And if you look at what Wendell's
6:22
up to on F41TEX, that's exactly what he's
6:24
doing. He's putting some like AMD Epic
6:26
server chip in there and balls
6:29
to the wall faster than VME it can
6:31
possibly be. It's kind of nuts. I
6:33
want that one. I want that one. So
6:35
this got me thinking about, you know,
6:38
for $800, let's review just
6:40
the chassis on its own for a second.
6:42
For $800, what else could I buy?
6:45
And the obvious choice
6:47
for me at least is the Rosewill LSV4500U,
6:51
which is a bit deeper and a bit longer and it
6:55
only has 12 drive bays
6:57
with the hot swap loading out the front
6:59
rather than out the top like this one. So I
7:01
mean, that's kind of a wash. Like if you want to load
7:03
the hard drives in and out the front of the case without having
7:06
to pull it in and out on rails, maybe
7:08
you consider that a pro. If you like
7:10
that kind of thing, then the 45 drives with 15 versus 12 is a pro for
7:15
that one. So that's really a wash
7:17
for me. The 12 versus 15 slots. I
7:20
don't know. I think I'd probably rather have the 15 drives
7:23
and then I don't also have to put them in sleds. Yeah,
7:25
that is the thing. I mean, the drive
7:28
bays at the front of this thing are tool-less. So
7:31
that's pretty cool. But you know, the other
7:33
thing I think about is five years ago when this CPU
7:35
was birthed into the world, hard
7:38
drive sizes were 8, 10, 12-ish
7:40
terabytes. And so 15 slots
7:44
back then was, air quotes,
7:47
only 150 terabytes, which
7:49
is still a hell of a lot of terabytes. Now, 20
7:53
terabyte hard drives are $300
7:55
a pop, which, okay, is not cheap, but
7:58
in terms of a rational purchasing decision,
8:00
you know, you could quite easily put 300 terabytes
8:03
in this box now in that same
8:05
density, which is if
8:08
you're using 300 terabytes at home,
8:10
I'd love to know what for
8:12
if it's a totally legal application.
8:16
That's all I'll say. Oh, I could see
8:18
it. I could see it. Yeah. These
8:21
days, especially with large models and, you know,
8:23
somebody wanted to bring other media for a media
8:25
production company. I could see it. I
8:28
mean, again, you're talking this is a device that lives
8:30
between the consumer like
8:33
the NAS for consumers and
8:35
the enterprise, right? This is
8:37
a 15 drive bay system. And
8:39
in the case with the fully loaded one, it comes with a 2018 processor. But
8:42
that kind of makes sense if you're coming from an enterprise design
8:45
standpoint. So it's for very serious folks.
8:48
And an $800 starting entry point
8:50
for just a chassis and the backplane for
8:53
people that are pretty serious seems
8:55
about right when you can put 15 drives in there and you
8:58
can put your own motherboard in there if you want and your own CPU.
9:01
Yeah, I mean, that's that's kind of nice.
9:03
It is a bit expensive, but it sounds really nice for
9:05
people that are serious. It is absolutely
9:08
a really nice bit of kit. It's
9:10
a premium bill quality. And
9:13
I've got no complaints about it. Really. I
9:15
just I just wish that it was five
9:18
or six hundred dollars, not eight hundred.
9:20
Oh, yeah. That would be a totally different equation
9:23
for me because at that price
9:25
point, it becomes the de facto like
9:27
standard unit of deployment for home
9:30
labbers and folks looking to build on raid
9:32
servers and and that kind of thing. They came
9:34
out with the HL-12. You know,
9:37
the HL-12, 12 bays, 500 bucks,
9:39
because I don't really know if it's the slots you're after.
9:42
It's you want something that's quality, right? You're going
9:44
to put this in your home. Craftsmanship. You're going to
9:46
have your home pictures on there. Right. Craftsmanship. I
9:49
think that counts for something. It does, which
9:51
leads me to a little bit of a
9:53
software experience out of the box that I had some issues
9:55
with too, unfortunately. So the image
9:57
they shipped to me had Rocky Linux. the
10:00
SSD and the
10:02
first thing I did was try to log in as
10:04
you can imagine. So first of all I had
10:06
to look for the IPMI password
10:09
which was printed on a tiny little sticker
10:11
on the motherboard. Their documentation unfortunately
10:14
told me it would be printed on a white sticker on the side
10:16
of the case that wasn't present so I had
10:18
to go spelunking on the internet and find
10:20
that Super Micro Motherboards as
10:22
of 2019 put this unique password
10:25
on the motherboard. And I spent
10:27
a while trying to get that
10:30
password in with the correct username of
10:32
admin and then realising that admin had
10:34
to be all caps. It
10:37
was case sensitive so it literally took me half
10:39
an hour to turn it on. That's on me,
10:41
some of it, you know, but some of it's on the documentation
10:44
too. Once I then got the system
10:46
booted up I was presented with
10:48
a GNOME login screen and I was like,
10:50
ooh username 45 drives. So I look in the documentation
10:53
for the HomeLab server which
10:56
is like 100 pages long. I had
10:58
to get to page 61 before I
11:01
found out what that username and password was
11:04
and it was wrong. It told me the username
11:06
was root and the password was correct
11:09
but it was for the user 45 drives
11:11
not the user root so you know some teething
11:13
issues there and nothing major,
11:16
just some small niggles to fix in the documentation.
11:19
Then I loaded up the terminal once
11:21
I got logged in and thought cool right time to update
11:23
packages as you do like a reflex after
11:26
you've used Linux for a long time. Absolutely.
11:28
And I had to do dash dash skip broken
11:31
dash dash fix something like
11:33
I had to pass three different parameters
11:36
to ignore all sorts of shenanigans that was going on
11:38
with DNF in order to fix a
11:40
bunch of out of the box broken packages which was a
11:42
bit disappointing. Hmm that
11:45
is. I'm not you
11:47
know I'm not sure if I'm actually even just thrilled
11:49
with the choice of Rocky. I do again I can see
11:52
it from the enterprise perspective. Well I think
11:54
they ship Rocky because they can do cockpit
11:56
modules and they spent a bunch of time
11:58
on their Houston UI. which is actually
12:01
based on top of cockpit and they've shipped
12:03
some custom modules in there which let
12:05
you see which drive is in which physical slot
12:08
on the system Some
12:10
proper system integrator level stuff, which
12:12
is brilliant. That's exactly what I want to see at this
12:14
price point And then the
12:17
only final Niggle I had
12:19
was I loaded up Firefox because
12:22
it has a desktop for some reason this server comes pre-installed
12:24
with a desktop Whatever, that's fine
12:27
And Firefox had someone else's browser
12:30
history in it Presumably the tech that was setting
12:32
up the server like it was looking at super
12:34
micro fan control and a bunch of other stuff that
12:36
They googled and gone through the github link and downloaded
12:39
the script and I was just thinking
12:41
to myself That
12:43
cheapens it a little bit because I would have expected
12:45
them to have a pre-made Golden
12:48
image with all of that stuff removed and you know,
12:51
I come from the enterprise world with red hat for a long
12:53
time Where building those kinds of experiences
12:56
out of the box experiences is exactly
12:58
what tools like satellite and ansible have been
13:00
doing for? Years and years and years
13:02
and I just I wonder why that wasn't
13:04
done here. It was a custom bespoke
13:07
edition Just for me. I Would
13:11
imagine too like that image is probably gonna be something
13:13
they're developing quite a bit. They'll be probably
13:15
updates their shipping Yeah, but knowing
13:17
you I don't I don't really see you wanting to run rocky in
13:19
production Are you planning to keep rocky install
13:22
on there? Are you gonna try going with something else? I
13:24
want to try putting nix on this bad boy There
13:26
you go. I actually want to do a full-on nix
13:29
OS server One of the guys
13:31
I work with has been showing
13:33
me some really ninja level stuff
13:35
with nix And system d n
13:37
spawn containers and how he's configuring
13:39
that with a like a helper script as part of his
13:42
flake setup I'll get in some details
13:44
into that over the next few weeks months because
13:46
it's it's pretty complicated and I'm still learning
13:48
it myself but it looks amazing
13:50
and Isolated containers
13:52
on the host as part of a nix declarative config
13:55
yada yada yada. Yeah, it's
13:57
pretty cool stuff I'm feeling it. That's what that's why
13:59
I want want to do our server here at the studio as
14:01
Nick's. We almost went Proxmox and then I started
14:04
thinking about it and like, wait a minute, we've missed
14:06
the window Chris, it's mid November. We should have done Nick's
14:08
November. I know,
14:11
I know. Nick's our new year maybe. But
14:14
so, I mean, it really should have no problem, right? It's
14:16
a standard PC, it's an
14:18
X86 box. Yeah. It should just load right
14:20
on there. It's just a Xeon box with a
14:23
CPU and memory. It's
14:26
just a Linux box, right, at the end of the day. I
14:28
know it's early but I mark my
14:31
words, in a couple of years, hardware
14:33
vendors like 45 Drives
14:36
and 45 Homelab and I also
14:38
think iX Systems, they
14:40
will realize that
14:42
to achieve the level
14:44
of quality and reliability they want,
14:47
it will be in their best interest to
14:49
ship this hardware with Nick's OS. I had
14:51
an absolutely fascinating conversation
14:54
with an individual in El Salvador who's shipping hardware
14:56
to stores and by a
14:58
process of elimination, they started with Debian
15:01
and went through everything else and by a process of elimination,
15:03
they're now using Nick's OS because of its quick
15:06
ability to deploy, recover and do all that stuff
15:08
that they need. I think a lot of hardware vendors
15:10
are going to get there. So, you know,
15:13
you'll have to keep us updated on what it's like to run it on this machine.
15:15
Well, I absolutely will. So, I think the way I'm going to split
15:18
my Homelab up now is I'm going to keep
15:20
my existing server with the i5-8500
15:23
CPU in it and Quick Sync. I'm
15:25
going to keep that for all the mundane sort
15:27
of media serving tasks that I have in the house
15:30
and then this one is going to be straight
15:33
up like business suit and tie, you
15:35
know, photo storage, next
15:37
cloud, that kind of stuff that, you
15:40
know, I'm going to throw a few hard drives in there
15:42
and do ZFS, all that kind of stuff. So, I think
15:46
for me it'd be nice to have that separation of duties
15:48
as well between like a serious server and
15:50
a fun server. Do I need it? No,
15:53
but it is fun as hell. Alex,
15:55
I know often when you're reviewing these
15:58
kind of units, energy... consumption
16:00
comes to mind for you. Have you measured this yet? Now
16:03
whilst I don't have 15 hard drives to load
16:05
this thing up with what I can tell you
16:07
is what the Standby power consumption
16:10
is because this thing has a BMC built into it for
16:12
proper IPMI because it's a super microsurfer
16:14
motherboard So just turned
16:17
off plugged in with just the BMC
16:19
on is 23 watts Which
16:21
if you consider that is already more than most
16:24
of those small form factor like 1 liter
16:26
PCs on its own True
16:28
gives you an idea of things but
16:30
better than those Dells that
16:33
we have plugged in Yeah,
16:35
you can't throw two terabytes of RAM
16:37
into a micro PC Can you I mean the more we talk
16:39
about it? I really see how this fits between like a one liter
16:42
PC home lab machine and also
16:44
like a huge used like super micro
16:47
Or old Dell or HP rack unit just
16:49
powered at rocky Linux just at the desktop
16:52
No hard drives installed nothing like that
16:55
roughly roughly 120 watts It's
16:58
a Xeon at the end of the day and that has Some
17:02
ramifications with with energy usage. So if you
17:04
compare that to like the i5 8500 for example, that's drawing 10 to 15
17:09
watts for idle. Okay, you've got a bunch of hard drives
17:12
in that box too But that's again,
17:14
we're be really nice to have click sync in that CPU
17:16
so you can throw a few more tasks out Yeah,
17:19
now it's a server. So they've put server grade
17:21
fans in it The CPU itself
17:24
doesn't have any fans on the heatsink So
17:26
that should give you an idea of the level of noise that
17:28
you can expect from the other fans in the
17:30
chassis, too You could legitimately
17:33
condition the air in your house with
17:36
this thing If you put a HEPA filter on the front
17:38
of it, the airflow is quite impressive So a
17:40
big thanks to 45 drives for sending that unit
17:42
over for review and you'll be hearing more about that in the
17:44
upcoming episodes That's
17:46
a great way to support the show you get a hundred
17:49
devices and you can see that tail scale truly is simple and secure
18:00
but we love that it's built on WireGuard. It'll
18:02
get up and running in just minutes. TailScale's
18:04
so great when you're traveling. I
18:07
mean, it's fantastic if you just work from home and
18:09
you wanna have a secure connection
18:11
to maybe a VPS and something that's running on your local
18:13
machine and a phone. You can kind
18:15
of create your own mesh network across all those
18:18
different networks. That's really handy. But
18:20
man, oh man, when I'm traveling,
18:22
oh boy is it great. Knowing
18:25
that every photo I was taking, every contact
18:27
I was saving, everything was being synced securely
18:30
over TailScale to my next cloud box, that's
18:33
a level of peace of mind that I just haven't
18:35
really been able to describe before when I'm joining like an
18:38
airport Wi-Fi in El Salvador. I
18:40
just love knowing that all my information's
18:43
protected by TailScale. And TailScale
18:45
Traverse Double NAT like it's nothing.
18:48
And when I'm switching between networks, going between
18:50
weird cellular providers I've never heard of and
18:52
onto an Airbnb Wi-Fi, TailScale
18:56
never missed a beat. My connections
18:58
between my servers, my computers, my mobile
19:00
devices, even some of my
19:02
appliances was just bulletproof. I
19:04
was managing my power input on my
19:07
Victron system using a subnet
19:09
router on my LAN and my RV connected
19:11
over cellular. And I'm doing
19:14
it all from my phone while I'm traveling.
19:17
It really is a great resource too for moving data
19:19
back and forth. They have TailScale Send which made it really
19:21
easy to copy files back and forth. If you wanna
19:23
share resources with folks, you can either integrate
19:25
it with your enterprise ACLs or
19:27
if you're just an individual, some friends, they got a system that
19:30
works for you. It'll completely
19:32
work when you're separated by firewalls and subnets.
19:35
It's just, it really is the chef's
19:37
kiss of networking. It's where I wanted to
19:39
see WireGuard go when I first heard it get described
19:41
way back in the day when it was just a twinkle
19:43
in the colonel's eye. From arch servers
19:46
to mobile devices, you've gotta try it out, deploy
19:48
it, and see how fantastic it is to build your
19:50
own private mesh network. Just
19:52
go to tailscale.com slash self-hosted.
19:55
You can try it for free for up to 100 devices, really
19:57
make sure it works for you, and it's a great way to
19:59
support the show. So it's tailscale.com
20:02
slash
20:02
self hosted
20:05
Well two weeks ago. We talked about the verge cast
20:07
and their big self hosting Podcast
20:09
episode and in that they interviewed
20:12
Alex who is the lead developer for the image
20:14
project now image is Ostensibly
20:17
a Google Photos self hosted
20:19
replacement or at least that's what it's trying to be So
20:21
Chris and I sat down with him to talk through
20:24
the future of the project and the roadmap and also,
20:26
you know Open source sustainability and that
20:28
kind of thing So we
20:30
finally got Alex on the show the developer
20:32
of image and it's kind of great timing
20:35
because version 1.8 6 just
20:37
came out and Alex welcome
20:39
to self hosted and sorry we haven't had you on the
20:41
show before now. No worry Thank you for
20:44
having me is the pleasure to be here to talk
20:46
to you guys and to talk about image Well,
20:48
the new release came out just a couple of days ago as
20:50
we record. It's got some nice features
20:53
I think specifically like around sharing right that
20:55
is correct Yeah, so
20:57
this new release we add the unified
21:01
Sharing for partners. So
21:03
now if you have multiple partners, you
21:05
can enable that option so all
21:07
of the access of the partners always is
21:09
also showing on your timeline and This
21:12
make browsing Experience
21:15
a lot better so you don't have to switch to
21:17
the sharing tab and then click on each partner
21:20
to see the new Media
21:22
but now just in your timeline But
21:24
there are a lot of caveat as this is the
21:26
basic version. The searching is
21:29
not including the partner media
21:31
yet
21:32
and
21:33
The only thing that share on that
21:36
media is the the location
21:39
For now, that's fantastic. Because if you
21:41
think about how most people actually use a photo
21:44
solution, right? My wife is
21:46
taking pictures of my kid I'm taking
21:48
pictures of my kid and we probably both want
21:50
to see pictures of our kid, right?
21:53
And we say it's a family unit thing and
21:55
a lot of the self hosted solutions
21:57
to date don't really take
21:59
that much multi-user within a
22:01
single household, a single family into
22:04
account. So I'm absolutely thrilled to hear that you guys
22:06
are. Thank you. And I think this
22:08
is coming from a lot of my
22:11
own use case as well, because scratch
22:13
the own, scratch your own itch. That's the best way to
22:16
develop software, in my opinion. Yeah. And then we
22:18
add the new map tire server to moving
22:23
away from using the free tire
22:25
server from OpenStreetMap as
22:27
the usage policy. I
22:30
thought that was really funny when I looked at it, because
22:33
you were like, well, when we were just a small scrappy
22:35
project, we could get away with using it for
22:37
free. But now we're kind of a big deal. So
22:40
we kind of need to do this properly by the book.
22:42
Yeah. The community, there
22:44
are some concerns that
22:47
asking us that we should move away as soon
22:49
as possible to not
22:51
overuse their free
22:53
resources. And we take
22:56
this change and we move away to a
22:58
different map tire server and
23:02
it's also giving us the opportunity to build
23:04
our own layers. So to
23:06
make it look slicker on the
23:09
dark and the light mode.
23:11
I think it does. I think it does.
23:14
So I want to talk about the audacity
23:16
of the project for just a moment, because the idea
23:18
of trying to replace Google Photos kind of seems like
23:21
an impossible task. And yet version
23:23
one dot eight six getting pretty
23:25
far now with face recognition, sharing
23:28
support, the location.
23:30
When I was looking at my pictures from my trip, the
23:32
new map stuff is really slick. So you
23:35
are kind of in a transition phase now to
23:37
a big project, right?
23:39
Yeah, I would say so. Just
23:42
to look at this core
23:44
member,
23:45
because they kind of group the
23:47
new user coming in by
23:49
day. So when you look at it, you can
23:52
see it grow. And
23:54
when I first started this core, we might
23:57
have one or two new people
23:59
come coming in. the server every
24:01
day and now when you look at it, it
24:03
can reach like 20-40 people. And
24:07
then the star
24:09
on GitHub is also the indication
24:12
of how the project is getting more popular.
24:15
And also, more
24:18
often than I would admit that I would browse
24:20
the internet to see how people are talking
24:22
about image. And yeah, we've
24:25
seen a lot of chatting and
24:28
most of the girls are coming from a worth
24:31
of miles. Yeah, it's coming up online all
24:33
the time now though. I feel like I see it getting mentioned
24:35
every week at least by some different group online
24:38
saying they're using it to replace Google Photos. People
24:41
are recognizing it for a genuine
24:44
replacement. It's pretty exciting. I
24:46
just wanted to say congratulations.
24:47
Thank you. My goal is
24:50
to build something for myself and
24:53
find image maybe what
24:56
it is. You
24:58
and I have crossed paths a couple of times on Reddit
25:00
I think in the past and talked about your funding model
25:02
in particular in terms of a
25:05
sustainable model for an open source project.
25:07
I mean, we've seen Home Assistant with Nebuchadse for
25:09
example build a company around
25:12
an open source project.
25:14
I think
25:15
we've also discussed in cross paths that
25:17
something like this photo
25:20
solution is so big
25:22
in scope that you do almost need a company
25:24
to support it. You've obviously
25:26
done amazing work and you've got some fantastic contributors
25:29
to date but I wanted to ask you what the plans
25:32
are for the future of image as a
25:34
project and also as a sustainable project
25:36
in open source. So I think there are
25:39
one thing that I
25:41
want to not never pivot
25:44
off is to not making it into
25:46
some type of paywall like
25:49
some of the open source projects
25:51
starting out as open source and then they got funding
25:53
from outside
25:55
investor and then they got that money pressure
25:58
to eventually have to make money. of
26:00
pivot of the first original idea.
26:04
So I don't want
26:06
to do that for image at all. I want to make it as
26:08
something good, something people
26:11
enjoy using. So that is something
26:13
that we would never like to move away
26:15
from. The car team
26:17
has talked about maybe eventually after
26:20
the service, the app is more stable.
26:23
We could think of building a
26:27
hosting solution for other people. So
26:29
people that would like to use image but don't
26:31
want to host it themselves. So
26:35
I think that is the, this might
26:37
be the only way right now to
26:40
make it as a sustainable
26:43
business. But by not
26:45
making it as the main
26:48
income source at home, for
26:51
all of us, I think that's making
26:53
it as a sustainable project
26:55
that keeps growing. Because we don't
26:57
have to depend on it for our livelihood.
27:00
You still have your day job and you still
27:03
dedicate your time outside
27:05
of job in order to
27:07
build image. And it's a
27:11
joy to do that as something,
27:14
I think the whole core team really enjoy
27:16
because we have such a nice group to
27:19
chat about technology, code,
27:22
and just how to
27:25
beat Google. You
27:27
love to hear that, right? And
27:29
really, it's great that while you grow
27:32
the project and get it to that point where it's stable
27:34
enough that it could be a product, that you do have the
27:36
support and income of a day job. And
27:38
maybe one day those two things merge and
27:41
image becomes a bigger part of your income source.
27:43
Yeah, and that's what I
27:46
try to achieve is when
27:48
it gets to that point where I can
27:50
replace the income source of image
27:52
into my own, that
27:55
I can support my family and the growth of
27:57
my family, then that would be fantastic.
28:00
work on something you love, you make good
28:02
things for people, and then you also get to earn money
28:04
from it. Yes, that would be absolutely fantastic.
28:08
How could people support the project today?
28:11
Today, there is only donation
28:14
means to support the project besides
28:16
testing and reporting bugs to
28:19
help us develop the project
28:21
better. You can support the project
28:23
financially from GitHub sponsor
28:26
or from Leapro
28:29
Bay Pay or buy
28:32
me a coffee. I drink a lot of
28:34
coffee, so keep the coffee
28:36
coming. In
28:39
the Vergecast interview that we talked
28:41
about on the last Self-hosted episode, you're
28:44
now a super famous Big Deal developer
28:47
and the big time on the Vergecast podcast.
28:49
I wouldn't say so. You
28:52
talked a little bit about your architectural
28:54
decisions about splitting it out into microservices
28:57
and a bunch of Docker containers and stuff like that. Is
29:00
there anything you wish you'd done differently now
29:03
or are there any big changes coming down the
29:05
pike? Yes, so the goal is
29:07
to
29:09
consolidate some of the services
29:11
like the proxy container and
29:14
the web container to put that into the
29:16
server container to make it less
29:18
scary when people are looking at the Docker Compose
29:21
file because those can be done. But
29:25
when I started it, it makes sense
29:27
to split it up at that moment.
29:30
And now if we might have found a way
29:33
just today to make the
29:35
web not server-side
29:37
rendering in anymore while still
29:40
keeping that really snappiness when
29:43
using the app.
29:44
So with microservice architectures,
29:46
you have the potential for remote
29:49
workers and stuff like that. Correct. I
29:51
mean, I would love to see it where I could distribute the facial
29:54
recognition. So I just dumped, for
29:56
example, like three or four terabytes worth of photos
29:59
into image. in the last couple of months because
30:01
you guys are awesome and added external library
30:03
support, which is amazing. Mwah, chef's
30:06
kiss. Thank you. But it
30:08
took like two or three weeks for it
30:10
to chew through all of those images
30:12
that I had. I've got lots of CPUs
30:14
in this house. It would have been great if I had an easy
30:16
way to distribute that task throughout
30:19
all my systems. Yeah. So
30:21
actually the idea of
30:23
breaking down microservices,
30:26
different containers, handle different things
30:28
is exactly to serve this use case. So
30:31
as of now, you can run
30:33
the machine learning on
30:35
completely different machine. And if
30:37
you put a frontend of load balancing
30:39
on it, then you can distribute it on
30:41
multiple nodes because we are
30:44
not accessing the file
30:46
on that container, but you
30:48
are sending the file
30:51
to those containers and then it would do
30:53
the referencing and then sending back the data
30:55
that it needs. The microservice
30:58
is a little bit more involved
31:01
because you would need to be
31:03
able to see the file system similar
31:06
to the server so that it can access and
31:08
do the transcoding stuff. But if you
31:11
have network mounted, you can also do
31:13
that already. Pretty typical distributed
31:15
network storage replication
31:17
problem there. Yeah.
31:20
But I think the
31:22
architecture has been designed
31:25
exactly for that
31:26
purpose. And we have a lot of people
31:28
that already do that. I think that will help you
31:30
guys as you move towards eventually
31:33
a hosted service. You will be able to pick the
31:36
cheapest VPS provider that week. Yeah. Do
31:39
you know it maybe? Yeah, we got a great
31:41
coupon code you can use. Well
31:44
I think for me and probably a lot of people
31:46
listening, I know Alex too, this is one of the tools
31:49
that has really been
31:51
the key to de-googling so
31:53
much of my private information. And
31:56
so privacy and security is super
31:58
important. I'm just curious to know your thoughts. around
32:01
enabling sharing but keeping people's
32:04
images safe and secure and how that kind of
32:06
works from a back-end perspective. Is your
32:08
question more about at
32:10
home? Yeah, like say if I wanted to share an image
32:12
with somebody, would there
32:15
maybe be like a back-end service like down
32:17
the road, some sort of intermediary that would be
32:20
the go-between to share the image so I wouldn't have to expose
32:22
say like my home machine or something? I'm
32:24
trying to think of ways of sharing images where I don't
32:27
I don't link someone to my image instance. That's
32:30
so challenging at the moment
32:32
but anything might
32:35
have a solution. We're
32:36
on video chat with Alex, dear listeners and
32:38
I can see the cogs turning right
32:41
now live as you're thinking what
32:43
if we had some kind of like remote cloud
32:46
you know. Yeah, that's what I'm wondering. VPS
32:48
like linked up or something you know that's
32:51
great idea Chris. Feature requests right
32:53
there. Turkey Days coming in and I'm thinking something
32:56
that I could like a page that would be generated
32:58
that I could share with family but I wouldn't have to link
33:00
them
33:01
they wouldn't all be necessarily loading it from my direct server.
33:04
I see so in that page you would talk
33:06
to the image server at home but that is
33:09
abstracted in the background and then
33:11
it would use the image API
33:13
in order to pull those just like a proxy
33:16
right just like a proxy pay. Eventually
33:18
this is something that we want is to build a plugin
33:21
system for the
33:23
people to build things around
33:25
an image. The API
33:28
is compatible with a lot
33:30
of the languages so we used open
33:33
API so it can generate
33:36
SDK for Python for JavaScript
33:39
for Rust if you want
33:41
for anything that you could think of right and
33:44
then they can use that in order to talk
33:46
to their server the image instant
33:49
and now one of the we have one really
33:51
good tool I would like to mention here is called image
33:54
go is from Simulat
33:56
it has this court hand
33:58
handle
33:59
This is a CLI
34:02
tool. You point it to your Google
34:05
Tech Al. You don't even have to extract
34:07
all the zip file that you download. It
34:10
unzip it and then it pars all
34:12
the metadata from that JSON file
34:14
that Google gave you. And
34:17
then upload everything into image
34:20
and create the album that comes
34:22
in your Google account. So it's just like one
34:25
click go. That's
34:27
so neat.
34:28
Yes. I love that. See, that
34:31
plugin system, if that really extends, it could be
34:33
a solution because people could build things around that. Because
34:35
when I really A-B, when I miss from Google Photos,
34:37
it's like a way to share photos and I'm not
34:40
like sharing any of my personal infra. But
34:42
something down the road, like a plugin
34:44
system, there's all kinds of ways to solve that. I'm
34:48
thrilled to hear that you guys are thinking more about that too. It's
34:50
going to be an exciting future as an image
34:52
user. Absolutely is. Now,
34:54
I have another question for you, Alex. Ever
34:57
since the beginning of the project, there's
34:59
been a banner in your documentation
35:01
and on your website that says, here be dragons.
35:03
Do not do anything serious
35:06
with this project because it's still under very active
35:08
development. Do you have any senses
35:10
or time scale as to how long before we can
35:12
actually start to air quotes
35:15
depend
35:16
on image? You know, in this court, we
35:18
have a funny way of saying that this is the any
35:21
question is a
35:23
prohibit question. We don't
35:25
talk about that here. Yeah,
35:28
fair enough. Don't commit to deadlines on air
35:30
in particular. I get it.
35:32
We put that because part
35:34
of it, this image is very active in
35:36
development, right? It's like just this
35:39
last release, the first time ever
35:41
we pushed out the first
35:43
release that if you don't, if you're not on
35:46
the same version, you cannot access
35:48
the instant from your phone at
35:50
all
35:51
because we changed the underground,
35:54
the underlying API. That's
35:57
the only way to resolve the difference is
35:59
to lock out. and lock back in on the same version.
36:02
Things like this could happen because we
36:04
are striving to go to the
36:07
industry standard for the
36:09
code base. If anything that we see
36:11
can be improved for the better future
36:13
of image, we will do
36:16
that. And since there are
36:18
many things that we still can improve,
36:21
we don't want to prohibit us to
36:23
backward compatibility
36:26
at the moment. But people
36:28
are striving to get to the point of feature
36:33
freeze or when the core
36:35
architecture is more
36:38
stable, we
36:41
would like to eventually
36:43
remove that
36:44
banner.
36:45
I think we all just see the potential of the
36:47
product and want to start using it. I
36:49
say product. It's not a product. Project.
36:51
And we just want to start using it today because
36:54
I'm real fed up of the
36:56
cloud nonsense and giving our machine
36:59
learning, overlord, model, so
37:02
much data. So I think that's
37:04
the only pressure really from us is that we kind
37:06
of wish it already existed because the pressures
37:09
that forced you to create it in the first place
37:11
are quite powerful. Yeah, very tempting.
37:14
They're tempting fruits, aren't they? Alex,
37:16
I'm going to keep using it even with that banner on there.
37:18
So there. And thank you so
37:20
much for creating it. And thank you to the entire team and everybody
37:23
that works on it and contributes. Absolutely. I
37:25
know a lot of us in the self-hosted community are just
37:27
extremely grateful. Yeah, I would like to shout out to the team
37:29
too. Everybody in the team
37:32
is great. They come from different backgrounds.
37:34
We have students from Europe. We have people
37:37
working in the application security, some
37:40
11x developer.
37:42
So the whole team is great.
37:44
And people are very
37:47
welcoming and willing to teach
37:49
each other. So I want
37:52
to keep this mindset growing in order
37:54
to attract more contributors
37:57
and just to make it a good place to be.
37:59
not just from using the application, but from developing
38:02
the application as well. And so if you'd like to contribute,
38:05
you can go to github.com and image-app,
38:07
I believe, is the place to go. You've
38:09
got a couple of hundred contributors already. A few more
38:11
wouldn't hurt, right? Yep.
38:15
So a big thank you for me and Chris for joining us on
38:17
the show today. And I'm sure this won't be the last
38:19
time that you check in with us. But until
38:22
next time,
38:22
thank you very much. Thank you guys. Thanks, Alex. Thanks,
38:25
Chris. You guys have a wonderful night.
38:28
So, I'm Chris
38:30
H. Linode is where we host everything in
38:32
the cloud. And now Linode is part of
38:34
Akamai, the Akamai. But with all the
38:37
tooling, the UI that we
38:39
like, the CLI that's super handy, all
38:41
that stuff that you can use to build, deploy, and
38:43
scale in the cloud, that's
38:45
still there. It's just now they're combined
38:48
with the power of Akamai's global reach.
38:50
They are the premier network out there. And
38:53
Akamai is expanding
38:55
their cloud services, expanding
38:57
tooling, but keeping things affordable,
39:00
reliable, and scalable
39:01
for users of any size, an enterprise
39:04
or an individual project. And something we've taken
39:06
advantage of in the last month is the expansion of the data
39:09
centers. They're expanding worldwide to give us more
39:11
access to more resources so you can grow
39:13
your project or your business. So
39:15
why wait? Go see why Linode is the only cloud
39:17
provider we recommend. Experience the power
39:20
of Linode, now Akamai. Go
39:22
to linode.com slash SSH. That supports
39:24
the show, and it gives you a $100 credit
39:26
so you can really kick the tires. So
39:28
go see how Linode, now Akamai, can help scale
39:30
your applications from the cloud to the edge.
39:32
That's linode.com slash SSH.
39:37
Well, at that time of year, Brent, you messaged me saying,
39:39
I want to buy some hard drives, and I couldn't
39:41
refuse the opportunity to sit down with you and
39:43
discuss hard drives. It's literally
39:46
my specialist favorite subject in the whole world.
39:49
Once a year, you get all giddy about getting new
39:51
hard drives, and I thought I'd join in this year. So here's
39:54
part one. Brent and I sat down and recorded
39:56
a few different segments to use over
39:59
the holiday season. and discussing
40:01
his journey into buying hard drives
40:03
and purchasing considerations, all that kind of
40:05
stuff. I'm going to put part two
40:07
in today's post show, which I believe, Chris,
40:10
we have a special offer for our members. It
40:13
is the Black Friday season after all. So
40:15
if you've been thinking about becoming a member and support the show and
40:17
you want to get that post show, use the promo
40:19
code Black Friday and you'll get $2 off a month for
40:23
a whole year of the SRE membership.
40:26
Just go to selfhosted.show slash SRE
40:28
and it's promo code Black Friday, all
40:30
one word. And then you get the post show and you get a
40:32
post show every single episode. So
40:36
all the way back in 2019, October 2019, four years ago, Brent,
40:39
you and I sat down with
40:44
Antonio, the developer for Merger
40:46
FS, and you professed to me in that
40:48
episode and public record,
40:51
no less, that you wanted to take
40:53
storage more seriously. And here we are four years
40:55
later and I think it's
40:57
about time we cashed that check. It's
41:00
long overdue, Alex. And I admit it
41:03
had been on my mind for four years previous to that
41:05
as well. So I am
41:08
in bad shape because we were talking about this before
41:10
we pressed record. And you said
41:12
to me, I've been buying drives with ZFS in mind
41:14
in pairs for like the last eight
41:17
years. No more than that.
41:20
Since my photography career required
41:23
it, which is, I had
41:25
been doing photography for 18, so I've got
41:28
paired one terabyte
41:30
drives and paired 500 gig
41:32
backup drives. Basically anytime
41:35
I'd buy drives, I would try to pair them because
41:37
I knew one day I would be smart enough to actually
41:39
do something about it.
41:41
And that day is today, is it?
41:43
I'm hoping it is, but my
41:45
level of success so far is challenging
41:48
at best. Well, okay. Let's
41:51
start at the beginning. You mentioned photography.
41:53
Now, I know that's a career path that was
41:56
huge for you previously, but now obviously
41:58
with your next cloud work. I would imagine
42:00
your data generation is a little,
42:03
you know, the velocity is a little slower on that side
42:05
of things. What kinds of stuff
42:07
are you looking to store at the moment?
42:10
Yeah, it's a good question. I think
42:13
most of my data, at least the
42:15
big stuff at this point,
42:17
is archival, to be honest. Like
42:19
it can just sit there and
42:22
won't get access very often, then it's a read-only
42:24
probably kind of scenario if ever.
42:26
Okay, so that leads me into my next question. If
42:29
it's archival, do you
42:31
genuinely need it to be always
42:33
on? Always accessible.
42:36
Hmm. I hadn't even considered that because
42:38
I had just pictured all of my storage being
42:40
in one box. But tell me
42:42
what you have in mind. If it's an archival situation,
42:45
it's not beyond the realms of possibility
42:47
that you've got a client ringing you up that you had
42:50
on the books 10 years ago that says, you know that shoot
42:52
you did in our mine, could you
42:54
find that particular, you know, shot
42:57
of Fred or whatever? I've
43:00
gotten those emails before. Yeah, I'm sure you have. And you think,
43:02
right, well if I just, you know, write down
43:04
what's on each hard drive, external hard drive
43:06
in a drawer, for example, that
43:08
could be a legitimate filing system for
43:10
that data. The issue
43:13
with that is Bitrot is a thing
43:16
and that keeping those drives just sat in a
43:18
hard drive, you know, they're not going to last forever.
43:22
So you could explore something like
43:24
Amazon's Glacier service,
43:26
which is a cloud based long term archival
43:29
situation. It's fairly cheap.
43:32
They do kind of hit you on the retrieval fees.
43:34
But if this is an archival situation only,
43:38
maybe it's a decent option. I hadn't
43:40
considered that at all as my primary
43:42
source of storage. I had only considered
43:45
cloud solutions as kind of a offsite
43:47
copy. I think mostly
43:50
because I have, I don't know
43:52
if I
43:53
trust them with the primary data.
43:56
Is that one way to look at it? But you're suggesting maybe
43:58
I have drives in a drawer and also... so
44:00
some data there, is that what you're suggesting? It
44:03
could be. I mean, the other option is you
44:05
just have all of that data running all the time
44:07
in your house, and it's always available on
44:09
top of ZFS so that you've got all the checksumming
44:12
and beautiful magic that ZFS gives you for data
44:14
integrity. And then you back
44:16
that up to Glacier. And the purpose for suggesting
44:19
that is it would reduce the
44:21
complexity for you somewhat in terms of having
44:23
to have a box at a remote house with, as
44:26
I found out fairly recently, having
44:29
a remote box in a different country or a
44:31
different side of the country even, even
44:33
if it's in a family house, it's
44:36
hard sometimes to get access to that. And the
44:40
economics of something like rsync.net
44:42
don't really work out. Seven
44:44
or eight terabytes in rsync.net is 70
44:47
or $80 a month. Oh,
44:49
I see what you're saying. Which pays
44:52
for a remote backup server in no
44:54
time at all. But Glacier
44:57
is, from memory, it's 10 bucks
45:00
for a similar thing or less
45:02
per month. Like I say, the
45:04
way they get you is on the retrieval. So if you try and do
45:06
an expedited retrieval, they'll charge
45:09
you $10 for a thousand API requests versus
45:13
just a standard rate of 3 cents per 1000 requests. Hence
45:18
the name Glacier. Because it's like,
45:21
only if the world is going wrong do you want to melt that thing.
45:24
Yeah, they've never actually published what the underlying
45:26
storage technology is. I suspect it's tape,
45:29
but we don't actually know. And so it would
45:31
make sense because I would imagine they have
45:33
those tapes in some kind of robot
45:36
automation system that can actually move
45:39
a physical piece of media in and out of
45:41
a specific location. And
45:43
if you're doing an expedited recovery, that probably
45:45
means some S hit the fan somewhere. And
45:48
you need that stuff back pretty quick. All
45:51
right, so let's leave Glacier for a second
45:53
and talk a little bit more about your local
45:55
hardware.
45:56
Desires, you sent
45:58
me a list of all the hard drives. you've got and let me just
46:01
let me just read this to the audience because it's
46:03
it's quite glorious I think you're going to enjoy this
46:05
is it really yeah I love I liked
46:07
it there is a Western Digital 8 terabyte
46:10
drive brackets shucked a
46:13
Western Digital red 4 terabyte drive
46:15
a Western Digital caviar black 2 terabyte
46:18
drive 3 seagate 1.5
46:21
terabyte drives 2 Western
46:23
Digital caviar 1 terabyte drives
46:26
and 2 500 gig drives
46:28
how many is that one two three four five
46:30
six seven eight nine ten ten drives
46:33
Brent that's quite a lot
46:35
unfortunately this doesn't include any of
46:37
the two and a half inch like external drives
46:39
that I've been carrying on all my travels to
46:42
do you know local backups as I go so
46:44
there are a couple more of those maybe we should throw
46:46
in the mix but they might not be useful in the setup
46:48
I don't know and then the next line in
46:51
in our little chat exchange was
46:52
but
46:53
the data duplication inefficiencies
46:56
galore etc so really
46:59
I have no idea quite how much storage I
47:01
have so if we add all
47:03
of that storage up dear listeners it comes
47:05
to approximately 21 and a half
47:07
terabytes but your guesstimation
47:11
on the use of that is somewhere
47:13
between 8 and 13 terabytes
47:16
is that right
47:17
yeah I think part of the challenge
47:19
here is that these
47:22
drives were never all available at the same
47:24
time and so they ended up being
47:27
generations of storage
47:30
for me you know someone fill up and then
47:32
they would go on let's call it long-term
47:34
storage and be replaced and
47:37
as they filled up depending on how
47:40
busy I was at the time or what it would just kind
47:42
of get slotted in and so the data is not
47:46
extremely well organized you know it's organized as
47:48
well as I could muster along
47:50
the way but because of that drive
47:52
swapping
47:53
there's also duplicates of stuff especially
47:56
with system backups and things like that there
47:58
are multiple multiple copies of that
48:00
kind of data and photography
48:02
projects. Sometimes on one set of
48:04
drives there's a project that was half finished
48:07
and on the next set of drives there's the whole project that
48:09
was finished or something like that. I
48:11
think if we approached
48:13
this intelligently and got rid of
48:16
all those duplicates, I would imagine there's way
48:18
less data than I think there is. But
48:20
to be honest, I'm just human and I have
48:22
no idea. You know what's interesting, I
48:24
think we've all been there. We've all found an old
48:26
hard drive in a drawer and gone, hmm, you know
48:28
what? I should back this up and then a couple
48:30
of years, elapse, and then
48:32
you find that same hard drive in a drawer and you go, you
48:35
know what? I should back this hard drive up. And
48:37
before you know it, you have that hard drive
48:39
in triplicate somewhere named slightly
48:41
differently, you know, every time. Well,
48:44
the thing for me is like,
48:46
this is just another iteration of my trying
48:49
to get that great solution. I mean, you've got
48:51
perfect media in the server, right? And
48:54
I've had various versions of
48:56
tons of hard drives in a box
48:59
and various versions of having it be
49:01
functional for me, but it never quite
49:03
got to the point where I felt really
49:06
comfortable about the data and that
49:08
it was, you know, being checked for parity and
49:10
that if a drive failed, that it wouldn't lose anything,
49:13
etc., etc. So I
49:15
still, despite my,
49:16
you know, knowing better and my efforts, I have
49:18
a long way to go, I feel like, which
49:21
is embarrassing, to be honest. Well, in
49:23
that case, my suggestion
49:25
is keep it simple.
49:28
Do a pair of disks
49:30
in a ZFS mirror and
49:32
then never think about it again. I mean, obviously monitor
49:35
them and all that kind of stuff. What I mean in terms of
49:37
you worried about like check summing
49:39
and all that kind of stuff and parity and then data
49:42
integrity, ZFS gives you all
49:44
of that out of the box because it doesn't
49:47
actually care about specific files.
49:50
ZFS cares about the blocks, the data
49:52
blocks, and it checks that if a specific
49:55
block is read from disk and doesn't
49:57
match the check sum it expects between the two mirrors.
50:00
it will go away and figure out what the right one is. And
50:02
then if it can't give
50:04
you the fully integral
50:06
data, it just
50:09
won't give you that piece of data at all. Which
50:11
sounds like a bug, but actually if you
50:14
ask me, it's a feature because it means that
50:16
you're never gonna get garbage out of a ZFS system.
50:20
The other nice thing about doing a pair of drives
50:23
like that is your unit of expansion
50:25
becomes really simple. And these
50:28
days you can buy 20 terabyte
50:30
hard drives for $300 each, which
50:32
isn't cheap granted, but
50:36
also for 20 terabytes of mirrored
50:39
storage, effectively 40 terabytes
50:41
of raw, 20 terabytes of usable.
50:44
That's actually not a bad deal at all. The
50:46
other thing about doing a pair is slot
50:49
density matters. You
50:52
might have, let's say four slots
50:54
in a server. So
50:57
let's put Brent in the shoes of five
50:59
years in the future Brent and this 20 terabyte mirror's
51:01
full. You think to yourself, well,
51:03
what can I do here? Well, you could just go and
51:05
buy another pair of drives of
51:08
whatever the in-vogue size is
51:10
that year. Let's assume it's 40 terabytes
51:12
in five years time. And
51:14
then suddenly you've more than doubled
51:17
your capacity, but all the old data is still
51:19
there in its old format for
51:22
you to move around between as
51:24
you see fit. The mental model,
51:26
it keeps it really simple to understand.
51:29
You know, that leads me straight into the
51:31
whole crux of the question that I have
51:34
today,
51:35
which feels like a bit of emergency. We're at
51:37
Black Friday season and a
51:40
bunch of drives are on sale and if you look at
51:42
them in Canadian dollars, I
51:44
have a potential to save like $200 per drive or
51:47
something with some of these sales. So
51:49
my question was, you know, you're suggesting
51:52
these, get the massive most drive
51:54
that I could possibly muster in my budget
51:56
and throw those as a pair
51:58
into the storage.
51:59
track that I have here. But it
52:02
gets me thinking about all these drives that I already have.
52:04
I've got an 8 terabyte, for instance. Can
52:07
I just buy an 8 terabyte drive and combine
52:09
it with that?
52:11
It seems like maybe
52:13
that would be far cheaper, and maybe that would
52:15
get me far enough where I could wait a year or two
52:17
and then get that massive pair
52:19
later. What do you think about the now versus
52:22
later question? The same logic
52:24
applies. Yeah. Get the biggest
52:26
hard drive your budget will
52:28
permit to pair with the old one.
52:31
Even if all you buy today is one 20
52:33
terabyte hard drive, for example,
52:36
pair that with the 8 that you've already got,
52:38
the 8 terabyte. Your mirror would
52:41
be an 8 and a 20. You'd be
52:43
wasting 12 terabytes today, which sounds
52:45
a bit stupid. But in
52:47
six months' time, when you come across another good
52:49
deal, and this just allows you
52:51
to do dollar cost averaging of hard drives,
52:53
I guess. You're buying
52:55
a hard drive now to make
52:58
the mirror and then in six
53:00
months' time, all you do is swap out the 8 terabyte
53:03
for the new 20, for example,
53:05
re-silver it, and suddenly then you've got the full 20
53:08
available to you. I had never
53:11
even considered wasting 12 terabytes
53:14
just for a couple months. It's a bit of a first
53:16
world opulent problem, isn't it? But
53:18
when you put it that way, when you're trying to
53:20
balance budget and maybe drive slots
53:23
and things like that, it
53:24
actually, in a strange way,
53:26
makes a lot of sense. But
53:30
it doesn't
53:31
make it easy to make a choice, does
53:34
it? No. The biggest knock
53:36
against ZFS really, versus something
53:39
like MergerFS, which you know I'm a great proponent
53:41
of,
53:42
is that
53:44
with ZFS you have to start with empty drives.
53:47
My question to you would be this 8 terabyte
53:50
drive, do you have somewhere to put that 8
53:53
terabyte drive whilst you build
53:55
the new array, the new
53:57
Z-pool?
53:59
It's a very... good point. I
54:01
will admit this 8 terabyte I also
54:03
may have
54:04
broke the connector on it so it's a little... Yeah,
54:06
it's easily done. So here's
54:10
the other thing right, we were talking about the data
54:12
you wanted to store and it's all kind of suit-and-tie
54:14
business-y data right? It's it's real
54:17
data that that cannot be replaced
54:19
stuff.
54:20
Yes.
54:22
Why not use your existing fleet
54:24
of mismatched hard drives to store
54:27
some slightly more replaceable stuff instead
54:29
using something like merger FS. Don't worry
54:32
about the parity on that and then have two
54:34
different tiers of data stores within the
54:36
same box using merger FS
54:38
and and that for one
54:40
side for the media and then ZFS for the suit-and-tie
54:43
data on the other.
54:45
I also hadn't considered that I'm
54:47
glad you're here with me. Yeah
54:50
and I like the idea of the peace
54:52
of mind of new drives because these drives you
54:55
know have
54:56
been around and just been sitting there as well
54:58
and they've traveled across the country in the
55:00
back of a trailer etc so my
55:03
confidence in these older drives
55:06
is also surprisingly low so the
55:09
idea of buying new drives feels I
55:11
don't know emotionally like a good thing but I
55:15
guess that brings me to another question of you
55:17
know if you go to somewhere like diskprices.com
55:20
to see what the best deal might be they just
55:22
by default show you a bunch of used drives
55:25
and those seem to have great prices
55:27
and they seem like enterprise drives and I'm just
55:29
curious your opinion of you
55:32
know if you shouldn't care about drives they
55:34
should just be replaceable then is that
55:37
a reasonable avenue to go down?
55:39
If you're running a data center yes. If
55:42
you are a man with less
55:46
than a dozen hard drives in his house
55:49
then each one actually matters at least
55:52
a little bit. You could argue that an off-site backup
55:55
makes that argument a little bit less relevant
55:58
but for every time a drive fails, it's
56:00
extra cost, it's extra cognitive load, it's
56:03
extra time watching and re-silvering, and
56:06
all that kind of stuff. So from my
56:08
perspective, you're
56:10
going to be saving probably 30% or 40% with
56:12
a used drive versus a brand
56:14
new one. But you just don't
56:17
know its history. There's
56:19
just no way to know how many times Linus Tech Tips
56:21
has dropped that particular drive, is there?
56:24
Oh, local drives. Yeah,
56:27
I guess you're right. The same applies. If
56:29
I'm not comfortable with the history of the drives that I have in
56:31
my own care, then I certainly shouldn't
56:33
be comfortable with the drives
56:35
that I know not the history.
56:37
There are certain things in a computer that I will absolutely
56:40
advocate buying used.
56:42
RAM,
56:42
definitely, because you can check that really easily. Really?
56:45
Yeah. CPUs, they almost
56:47
have only ever had one go bad
56:49
on me. Motherboards, again,
56:52
they either typically work or they don't, as long as you check
56:54
the CPU pins very carefully. Power
56:56
supplies, typically, they're
56:59
pretty reliable, too, as long as they've got all the original
57:01
modular cables with them. You don't swap them
57:03
out with a different manufacturer, because the pinouts can be
57:05
wrong, and then wrong voltages fry
57:08
things. Obviously, a case, as
57:10
well. Obviously, that doesn't really matter if
57:12
that's used or not. But hard drives
57:14
are the one thing, because they are so
57:17
delicate, and the tolerances are so small, and
57:19
they're so fragile.
57:21
I just don't mess about with them. Well,
57:24
that's great advice. Thank you, Alex. It feels like
57:26
now I just got to open my wallet, see what I get
57:29
shipped here this week.
57:32
45homelab.com, the HL15
57:35
is here. It's available for purchase. You
57:37
can get it in a bare bones thing, or you can get
57:39
it built up with a chassis and a backplane
57:41
and the PSU, ready to go, fully tested.
57:44
It's really up to you. Big, strong, fast storage
57:46
for the Homelab. Taking the enterprise
57:49
mindset, the lessons learned that 45
57:51
Drives has taken from all their years in the enterprise
57:53
and collaborating with the open source community, and
57:56
now building something for the Homelabber.
57:58
It comes with Rocky Linux. you'll have the ability to
58:00
install applications on it. Things like Image will
58:03
run fantastic on the HL-15.
58:06
And from what I hear, Alex has one arriving
58:08
soon, or maybe already arrived, actually. I
58:11
don't want to say too much, but chances are there's
58:13
going to be a review on the show. I'm really excited
58:15
because now we have something that I think
58:17
meets the bar, and it takes this
58:19
as seriously as we all do. So go try
58:22
it out. I think this is the perfect unit for anybody that
58:24
wants something robust that can grow with them.
58:26
It's the HL-15, and you can find it over at 45homelab.com.
58:31
It's here. It's based on your feedback, and
58:33
it's beautiful, big, strong, and fast at 45homelab.com. Well,
58:38
Alex, thanks so much for your insights. As always,
58:41
I
58:42
should just listen to what you say and just go and do
58:44
it. So thank you for that. Oh,
58:46
stop. But I bet you we've got a whole bunch of
58:48
booths we should go through.
58:50
We have a nice batch this week. We can't always fit them
58:52
all for runtime.
58:54
But Bamham 182 comes
58:56
in with 34,567 SATs using
59:01
Podverse this week. And they write, I
59:03
appreciated hearing about the backups as I'm building
59:05
a backup server. I've gotten an Elite
59:07
Desk 800 G3 SFF for $40 on eBay. That's
59:13
amazing. That's what I keep saying
59:15
about these one-liter PCs, man. They
59:17
are bargain of the century. And
59:20
it has two 3.5 MVME drives in it. I
59:23
bought the Easy Stores, which will
59:25
replace some drives in my main zPool. These
59:28
will go into the Elite Desk. If all goes as
59:30
planned, the box will go to a friend's house and
59:32
is off all the time. But it'll turn on at night
59:34
and sync the data through WireGuard and then
59:37
turn back off. I switched from PFSense
59:39
to OpenSense with all of the WireGuard
59:41
drama, and then from OpenSense to
59:43
NixOS because I became more and more
59:45
familiar with NixOS. Absolutely
59:48
no regrets so far. I love having full control
59:50
and not having things I don't need running 24-7. I
59:53
recently wanted to swap out a Dell Wyze I
59:55
was using for a T740. I
59:58
was able to get the T740 running. with a
1:00:00
very similar config, swap the boxes in it
1:00:02
without the family even realizing I had
1:00:04
done it. Nice job, B.
1:00:07
Nice. I have a question here.
1:00:10
Given these little one liter PCs and they just
1:00:12
sip power, why would you bother
1:00:14
turning them on and off at night like that? It's
1:00:16
probably not your power bill. It's not your power, right?
1:00:18
So even a little bit, still something, right? Yeah,
1:00:21
but I just worry about
1:00:24
mechanical hard drives, spinning up, spinning down.
1:00:26
Is that a consideration here? Or am I just
1:00:29
worrying for no reason? It is a consideration.
1:00:31
It adds significant complexity
1:00:33
to all of your monitoring as well. Also
1:00:36
the exit conditions of the backup scripts
1:00:39
itself, it becomes a consideration
1:00:41
to you. If the server's on all
1:00:43
the time, you don't have to have any of those considerations.
1:00:46
You just can assume it's always on.
1:00:49
And if ZFS tries to do a replication, it will
1:00:52
say, well, you're already doing a replication dummy. Don't
1:00:55
try and do another one. It's in the middle of one and it shuts
1:00:57
the computer down. And then
1:01:00
let's say you just took a trip, hypothetical
1:01:02
example, and you're dumping 100 gigs worth
1:01:04
of pictures onto your local
1:01:07
system. That could easily take
1:01:09
a couple of days or more to upload to the remote
1:01:11
site. And in that case,
1:01:14
you've got to plan for that one-off
1:01:16
eventuality, that unusual eventuality. But
1:01:18
you've still got to plan for it. And like I say,
1:01:21
it just adds complexity to the whole situation.
1:01:24
And reading the rest of this comment, you're
1:01:26
clearly a gentleman that appreciates the finer
1:01:29
things in life by ditching any
1:01:32
non-declarity of OS. So
1:01:35
look at us go. I
1:01:39
love it. That's right, the gentleman's OS. I
1:01:42
think it's a pretty solid setup. I totally get
1:01:45
turning it on and off to save power,
1:01:47
although I have a similar setup here. I have
1:01:49
my main workstation
1:01:51
on a
1:01:52
Zigbee Smart Plug. When
1:01:55
I arrive, I have an automation that turns
1:01:58
on the Smart Plug and my BIOS is set. to
1:02:00
restore power when the power
1:02:02
state comes back. And that's worked
1:02:04
for six months. And then when I went on my trip
1:02:06
and came back, the smart plug turned on. And
1:02:09
now every other time, the PC doesn't restore
1:02:11
power state. Just for some reason, still
1:02:13
set in the BIOS, I haven't changed a thing,
1:02:16
and it just doesn't restore power state. So that kind of thing
1:02:18
can happen too.
1:02:19
Now, Alex, this gets me thinking about, you
1:02:22
sent me a photo a couple days ago of
1:02:25
something you 3D printed for your one liter PC,
1:02:27
which to me on the surface seems ridiculous.
1:02:30
Oh, yeah. But
1:02:32
can you maybe give us more details?
1:02:35
I 3D printed a rack mount, a
1:02:37
one new rack mount for these little systems. Chris,
1:02:39
I'm going to just see your listeners. Here we go. Oh,
1:02:43
my goodness. This is amazing. So I've
1:02:45
got one of these little Dell boxes, one of these little
1:02:47
PCs in a rack mount
1:02:49
system now.
1:02:51
That is so cool.
1:02:54
And then obviously, a 3D printer can't fit 19
1:02:56
inches, which is a standard rack unit width. So
1:02:59
you split it in half, put some screws in the middle,
1:03:01
and suddenly you've got a 19 inch wide
1:03:04
one unit for two of these things. You
1:03:06
put two of them in there, one, yeah, and one. Oh, my gosh.
1:03:09
There's quite a few if you look on like principles or Thingiverse
1:03:11
or something like that. There's quite a few people that sort
1:03:13
of stack them on top of each other like this, and you can
1:03:16
fit an eight port switch in there
1:03:18
as well. So you can actually literally have like a five or
1:03:20
six half width U
1:03:23
rack, like a mini rack almost,
1:03:25
like a desktop home lab rack, just
1:03:27
3D printing. I need that for the ODROID.
1:03:30
I have yet to find a case I am fully satisfied
1:03:33
with, with the ODROID. And I just need a rack
1:03:35
unit that the top is exposed. That's
1:03:38
really great. Just buy a bamboo lab printer, dude, and be done with
1:03:40
it and get it over with. All
1:03:42
right. Johnny Castaway comes in with 26,235 stats using podverse. Johnny
1:03:47
Castaway? Isn't that great? And that's a great name.
1:03:50
He's a member. That brings me back. That
1:03:53
brings me back. Longtime listener. And
1:03:56
he says thank you for contributing to everybody who makes
1:03:58
my eight hour weekly commute. We
1:04:01
are the we are the commuters friend. He
1:04:03
says I was once at peace with arch then I
1:04:05
tried Nick's OS gave up
1:04:07
many times returned back to my old arch slippers,
1:04:10
but Nick's kept taunting
1:04:12
me like a beast Nick's has been tamed
1:04:14
now, and I think it's here to stay He says
1:04:16
my next endeavor is Nick's OS on the steam
1:04:18
deck any thoughts No,
1:04:21
do you want to mess up a good thing? Oh, it's a
1:04:23
UK. He also comes in from the UK Yeah,
1:04:26
all of this learning people must do these
1:04:28
days, right? Yeah,
1:04:31
it's funny because I was such a curmudgeon
1:04:33
when it came to Nick's as well and Chris brought
1:04:35
it up in the show and I
1:04:37
believe privately said to me Alex if you just listen
1:04:40
sometimes I identified Nick's years
1:04:42
ago Why couldn't we've done this in self? I
1:04:44
was like yeah, but I just didn't see it and
1:04:46
it's just one of those things like nobody
1:04:48
nobody can tell you how
1:04:51
good Nick's is nobody can Can
1:04:53
tell you how it's gonna change your
1:04:55
life You just have to experience
1:04:58
in it just has to click For you
1:05:00
and it's like that moment as a teenager
1:05:02
where your parents tell you you just don't understand
1:05:05
You're like well one day eventually
1:05:07
with enough experience you
1:05:09
realize just how powerful
1:05:12
that stuff can be You
1:05:14
know in Linux unplugged sometimes we get a hard time because
1:05:16
we're mentioning Nick's OS Far
1:05:18
more this year than well we ever
1:05:21
had But
1:05:22
it's just
1:05:23
because it solves so many issues
1:05:26
in a beautiful way I mean, there's that learning
1:05:28
curve, of course, but I
1:05:30
have heard just this week alone
1:05:32
like five to six different
1:05:35
people say Oh, yeah.
1:05:37
Thanks guys. You made me try Nick's OS now. I can't
1:05:39
use anything else and So
1:05:41
if you haven't tried it yet, I I'm
1:05:44
running it on this laptop in front of me and I was a curmudgeon
1:05:46
too It took me a while to come to it and it just like it's
1:05:48
just kind of nice It's kind of nice
1:05:51
another tool in the toolbox. I mean you look at
1:05:53
the the wider ecosystem
1:05:55
I mean the whole flakes adoption situation
1:05:58
is frankly. It's just a mess But
1:06:00
you've got companies like Determinate Systems with
1:06:03
Flake Hub coming along, pushing the agenda, and
1:06:05
taking the project by the scruff of the neck and saying,
1:06:08
Come on boys, sort this out. That's
1:06:10
exactly what it needs. And for me,
1:06:12
those signs are the green
1:06:14
shoots that I needed to know that the project was
1:06:16
going to be worth investing in in the long term.
1:06:20
Yep, I completely agree. Although I'll take the other side
1:06:22
just to be fun and say, Don't try it because it'll wreck you. Stick
1:06:24
with your distro. Stick. Don't change it. You'll
1:06:26
never be the same. The Bitcoin
1:06:29
Dad Pod comes in with 24,444 sats using Podverse. It
1:06:33
writes, Backblaze Cloud
1:06:35
is another remote endpoint for backups. I use Restick
1:06:38
and their S3 encrypted object storage. My
1:06:40
last bill was $4 for just two
1:06:42
terabytes of storage. Whoa.
1:06:45
That's attractive. Mm-hmm.
1:06:46
Noted. It says I've deployed Tailscale
1:06:49
at limited scale and I find it amazing. That
1:06:51
said, I'm concerned about ReLion on it
1:06:53
for as a third party now that it has
1:06:55
visibility into my network. They also seem like
1:06:57
an obvious point of regulation in a world of governments
1:07:00
obsessed with surveilling private online activity. Has
1:07:02
anyone ever tried to migrate a Tailscale set up to Headscale?
1:07:06
What are your thoughts on the potential risks? Well,
1:07:08
I can't speak to the migration to Headscale because
1:07:10
I completely trust the Tailscale
1:07:13
infrastructure. And here's why. If
1:07:15
you know anything about public key
1:07:17
infrastructure, you will know that that is
1:07:19
what WireGuard is based on top of. So the
1:07:22
idea that you provide a key, the remote server
1:07:25
authenticates based on that kind of stuff, and
1:07:27
it's very difficult to break that
1:07:29
level of encryption. Any
1:07:32
data that transits any Tailscale
1:07:34
asset is encrypted using
1:07:37
that methodology. But
1:07:39
the thing about Tailscale that kind of makes it magic,
1:07:42
and okay, yes, I work for them. Corporate
1:07:44
shill moment, hat on, is
1:07:47
most of the connections happen through the magic
1:07:49
sock. So they become peer to peer connections.
1:07:52
So very little data actually traverses
1:07:55
any Tailscale owned infrastructure. Any
1:07:58
point you can look at Tailscale, scale status,
1:08:01
look at all the nodes in your network, you can do tail
1:08:04
scale net check, which prints an analysis
1:08:07
of the local network conditions,
1:08:09
like all the local DERP servers, all that kind of stuff.
1:08:12
So when you are connecting two tail
1:08:14
scale nodes together, they initially
1:08:17
connect and report back through
1:08:19
the tail scale control server via
1:08:21
this DERP mechanism, D-E-R-P.
1:08:24
The only thing that's identifiable on there is, you know, like an
1:08:26
IP address, that kind of stuff, but frankly,
1:08:29
that's identifiable across many other
1:08:31
situations than just tail scale. The
1:08:33
actual packets, the actual traffic? Tail
1:08:36
scale aren't interested in that, because that would cost them money
1:08:38
to transit through their servers, all that kind of stuff. And
1:08:41
if you look at the free tier that tail scale offers too,
1:08:43
those particular users that are on the
1:08:45
free tier, you think to yourself, well, how
1:08:48
can that possibly make sense from a business perspective
1:08:51
for them to offer this thing for free? Well,
1:08:53
it's because no data transits tail scales network,
1:08:55
and they're still using something like tail scale funnel or something
1:08:57
like that, and it's relayed through a TCP proxy.
1:09:00
But that's from like hosting a very simple website as
1:09:03
a developer or something like that. So,
1:09:05
you know, in terms of trusting them, there isn't much
1:09:07
trust, they actually need to give tail scale.
1:09:09
The only thing I need to give them is that they're going to continue
1:09:12
to exchange my wire guard keys for me. And
1:09:14
that's it really, that's the only trust that I need to put
1:09:17
in tail scale as a company and
1:09:19
to continue to exist, of course. Yeah, and
1:09:21
I also probably could have done more to play
1:09:23
around with head scale. But part of
1:09:26
the overall solution that I find super attractive
1:09:28
is that the hosted component where they
1:09:30
do that sort of derp discovery, I guess, would
1:09:33
be the term, which I love, is
1:09:36
sort of the value added that I needed to actually
1:09:38
start using it. I kept having
1:09:40
one off wire guard setups that would
1:09:43
I would have for six months or nine months at
1:09:45
a time, maybe a year at most, then
1:09:47
I'd rebuild it and set up in a different way. And I have a different
1:09:49
way to, you know, maybe even try to manage
1:09:51
it. And
1:09:53
that just,
1:09:54
that worked. But it was, it's
1:09:56
like it's a, it's a, it's a, it's a tail scale is
1:09:58
a different level of a solution. Because it brings
1:10:01
all my devices into one mesh network
1:10:04
And I was never gonna get to that level with
1:10:07
anything that I had to kind of self set
1:10:09
up like that It's just it just what I was never gonna spend
1:10:11
the time And so that's kind of that
1:10:13
extra value they brought there is what made me put
1:10:15
it on everything You could argue that switching
1:10:17
solutions every six months is pretty secure
1:10:20
because by the time any script kiddies are cool Talk with
1:10:22
you you're on to the next thing. Yeah, that's true.
1:10:24
That's true Now we do have more boosted
1:10:26
in making to the show because we had ten total boosters
1:10:29
this week But we'll put a link to the boost barn. We
1:10:31
do read all of them, and then we save them for posterity
1:10:34
So thank you everybody who boosts in we stacked
1:10:36
three hundred and forty six thousand eight hundred and
1:10:38
four sats this week we really appreciate
1:10:40
that you can boost in by getting a podcast
1:10:43
app at podcast apps calm like pod verse
1:10:45
and Cast-a-matic and
1:10:47
fountain those are really popular in our audience and
1:10:50
if you don't want to switch apps We'll just get out beat get albie
1:10:52
calm you top it off and head over to the podcast
1:10:54
index and boost and we'll have links To that and
1:10:57
also thank you to our members once again. Don't forget. We got that
1:10:59
black Friday So two bucks off
1:11:01
a month for a year. We use promo code black
1:11:03
Friday You can
1:11:06
find me at Alex dot Katie said me
1:11:08
got a bunch of links to the various places But
1:11:11
I'm on the internet over there one
1:11:13
place you can find me as Linux unplugged Linux
1:11:15
unplugged calm
1:11:17
Oh bunches shenanigans happening over there,
1:11:19
too
1:11:20
Yeah, you can find me over there sometimes not
1:11:22
every week though sometimes. It's evil Chris You
1:11:24
just have to figure out if it's good Chris or evil Chris, but this
1:11:26
is good Chris for sure Although if it was
1:11:29
evil Chris I wouldn't tell you but they're both tweeting
1:11:31
at Chris LAS on weapon X and the show
1:11:33
is also at self hosted show if you Want to follow that
1:11:35
over there? Thanks for listening everybody. That
1:11:37
was self hosted not show
Podchaser is the ultimate destination for podcast data, search, and discovery. Learn More