Podchaser Logo
Home
110: Google Photos Replacement

110: Google Photos Replacement

Released Friday, 17th November 2023
Good episode? Give it some love!
110: Google Photos Replacement

110: Google Photos Replacement

110: Google Photos Replacement

110: Google Photos Replacement

Friday, 17th November 2023
Good episode? Give it some love!
Rate Episode

Episode Transcript

Transcripts are displayed as originally observed. Some content, including advertisements may have changed.

Use Ctrl + F to search

0:00

Once again today we're joined by Brent.

0:02

Hello Brent, how are you? Hey Alex, I'm good.

0:04

How are you guys? Well we're fantastic now

0:06

that you're here. It's always nice to have a little maple syrup flair. And

0:09

you know what? I just got back from El

0:11

Salvador and there was no maple syrup

0:13

to be had. Except for at Wendy's.

0:15

But you don't want their maple syrup. Makes sense. You

0:17

made it back in one piece, huh? I did. I did.

0:20

I'm feeling... Towards the end of the trip I was actually feeling pretty

0:22

good. I didn't really want to go. It's a

0:24

good school figure, right? But that's

0:27

just sort of how it goes sometimes.

0:29

But I'm glad to be back because we've got a packed

0:32

episode. Not only do we have a special guest

0:34

coming up, the core developer behind Image. But

0:37

Alex, you got some new gear

0:39

in the Homelab. It's been a veritable

0:42

circus of UPS drivers the last couple

0:44

of weeks. Not only did I get a new 3D

0:46

printer, I got one of the Bamboo Lab ones. Which

0:48

is awesome, by the way. I

0:51

also got my 45 Homelab

0:53

server review unit sat on the desk here

0:56

behind me.

0:58

Big and chunky boy on the desk.

1:00

It's like a 4U box. I'm

1:03

trying to

1:04

describe to you through the

1:06

medium of audio just how big this thing is. It's

1:09

like one of those fractal, defined

1:11

size cases. Just flipped on its side and designed

1:14

to go in a rack. They do ship

1:17

rubber feet to go with this thing if you want to stand

1:19

it upright vertically. One

1:21

of the things I really like about it is on the front

1:23

there's a pair of captive thumbscrews. To

1:26

get inside, they're on the front

1:28

of the case. Imagine it's in a rack situation.

1:31

You've got two captive thumbscrews that you twiddle.

1:35

Once they're loose, you just push the top of the case

1:37

back a little bit and pop it up. Then you've got access

1:39

to all of the internals of the case. They

1:41

sent me the fully loaded unit. The

1:43

one with the 6-core Xeon CPU in it, 32GB of RAM, 1TB NVMe

1:45

SSD.

1:50

Also, the 15 pre-wired

1:53

drive bays, which is just fantastic.

1:56

In this case, the build quality is the

1:58

first thing that struck me.

1:59

It is absolutely flawless.

2:02

This thing is built like a tank, really

2:04

thick. I assume steel, metal

2:07

of some description, I assume it's steel. Really

2:09

thick, good quality machining, no

2:11

tool marks. The powder coat job is fabulous.

2:15

It's up there with System 76's

2:17

build quality from when we took a tour of their factory

2:20

a few years ago. I don't know about you Chris, but I'm feeling

2:22

all sorts of envy. I know, I'm thinking,

2:24

boy, this sounds like a pretty nice rig.

2:26

I actually do think I'd wanna have it standing out

2:29

on the table or something. Maybe I wouldn't want it in the rack. Within

2:32

arm's reach, like Alex, you just give it a slap to

2:34

prove it's actually there. Yeah, so you can give it a good smack, yeah.

2:37

So that's the aesthetics of it. Have you taken

2:39

it much further than that? Just first impressions I assume

2:41

so far. Well, the CPU that's in there is

2:43

a Xeon 3204 from 2018 with

2:47

a 1.8 or 1.9, the internet can't quite agree on

2:52

the core base clock of this CPU. 32 gigs

2:54

of RAM that's in there is ECC memory

2:57

and the one-tier SSD is an

2:59

NVMe one, as I mentioned. I

3:01

find it really interesting that they went with a Xeon CPU

3:04

and there's obviously some reasons why you would

3:06

do that. If you look at the PCIe lanes

3:08

that Xeon affords you, it basically

3:11

enables them to put the HBA

3:13

card directly onto the Super Micro motherboard

3:16

that ships in this thing. It's an

3:18

X11 SBH motherboard. I opted for the one

3:20

with the SFP plus 10 gig pair

3:22

of NICs in it. They do make a

3:25

standard RJ45 variant

3:27

as well. Alex, I heard you say 2018 in there. Does

3:31

that seem quite old these days? Yes,

3:35

and there is no way around that. It's a six-core

3:37

CPU with no media

3:39

encoding engines either. So there's no quick sync with

3:41

this chip. That single

3:43

decision led me down a bit of a path

3:46

of thinking who is this box for? Because

3:49

when we had Doug and Mitch on a couple of weeks ago, talking

3:52

about how it's big, strong, fast, stable

3:54

storage, all that kind of stuff, the

3:56

performance for just storage

3:59

is really... But

4:02

the minute I have a server on in my house

4:04

24-7, I'm looking at what else can

4:06

it be doing. I want to be doing image,

4:08

I want to be running image on it and doing a bunch of machine learning

4:11

tasks on it. I want to be doing Jellyfin

4:13

or Plex or whatever and doing a bunch of transcoding on it

4:15

maybe for remote

4:18

media streaming, that kind of stuff. And

4:20

I find the choice of a 6-core

4:22

CPU with a sub 2GHz clock

4:25

speed from 5 years ago peculiar.

4:29

I understand that the choice of motherboard

4:31

is probably what precipitated the choice of

4:33

CPU because

4:36

certain motherboards have certain features like HBA's

4:38

built into them like this one does and it supports

4:40

up to 2TB of RAM because it's a Xeon

4:43

chip. All this kind of stuff.

4:45

There's a lot of very valid reasons for

4:47

doing a Xeon. However,

4:51

if you're putting this in a home lab, it's in

4:54

the name. It's probably going in my home.

4:57

And for most people at home, you

4:59

are using a box with 15 hard

5:01

drive slots to store some kind of media,

5:04

probably. You might be a video

5:06

editor. I can't imagine that's a huge

5:09

use case, although certainly there will be

5:11

a bunch of YouTubers that use this as

5:13

their primary video editing

5:15

storage system. But I've got to imagine

5:17

that most people building these or using these

5:20

cases to build servers

5:22

are doing things like Unraid and TrueNAS

5:25

and Perfect Media Service, dare I say

5:27

it, all that kind of stuff. And so why wouldn't

5:29

you put a CPU in it that

5:32

has hardware media encoding?

5:35

It really confuses me as

5:37

to the identity of the product

5:39

a little bit. I bet it is predicated

5:42

on the motherboard. And I wonder if

5:44

just describing which motherboard and CPU

5:47

are coming in the system sort of solves this problem so then

5:49

you just know before you pull the trigger here. And

5:52

maybe some folks don't care about QuickSync. I

5:54

mean that is a thing, right? You can just buy the

5:57

chassis on its own, the case on its own, for...

6:00

about $800. So the fully loaded unit that

6:02

they sent me, which full disclosure I did

6:04

not pay for, the full unit

6:07

is $2,000. They sell a version of just the case

6:09

for about 900 and then the

6:12

case and the power supply for 900 and then

6:14

just the case is about 800. Right.

6:17

So you could, I guess, get that one and put your own motherboard

6:20

and CPU in. Absolutely, yeah. And if you look at what Wendell's

6:22

up to on F41TEX, that's exactly what he's

6:24

doing. He's putting some like AMD Epic

6:26

server chip in there and balls

6:29

to the wall faster than VME it can

6:31

possibly be. It's kind of nuts. I

6:33

want that one. I want that one. So

6:35

this got me thinking about, you know,

6:38

for $800, let's review just

6:40

the chassis on its own for a second.

6:42

For $800, what else could I buy?

6:45

And the obvious choice

6:47

for me at least is the Rosewill LSV4500U,

6:51

which is a bit deeper and a bit longer and it

6:55

only has 12 drive bays

6:57

with the hot swap loading out the front

6:59

rather than out the top like this one. So I

7:01

mean, that's kind of a wash. Like if you want to load

7:03

the hard drives in and out the front of the case without having

7:06

to pull it in and out on rails, maybe

7:08

you consider that a pro. If you like

7:10

that kind of thing, then the 45 drives with 15 versus 12 is a pro for

7:15

that one. So that's really a wash

7:17

for me. The 12 versus 15 slots. I

7:20

don't know. I think I'd probably rather have the 15 drives

7:23

and then I don't also have to put them in sleds. Yeah,

7:25

that is the thing. I mean, the drive

7:28

bays at the front of this thing are tool-less. So

7:31

that's pretty cool. But you know, the other

7:33

thing I think about is five years ago when this CPU

7:35

was birthed into the world, hard

7:38

drive sizes were 8, 10, 12-ish

7:40

terabytes. And so 15 slots

7:44

back then was, air quotes,

7:47

only 150 terabytes, which

7:49

is still a hell of a lot of terabytes. Now, 20

7:53

terabyte hard drives are $300

7:55

a pop, which, okay, is not cheap, but

7:58

in terms of a rational purchasing decision,

8:00

you know, you could quite easily put 300 terabytes

8:03

in this box now in that same

8:05

density, which is if

8:08

you're using 300 terabytes at home,

8:10

I'd love to know what for

8:12

if it's a totally legal application.

8:16

That's all I'll say. Oh, I could see

8:18

it. I could see it. Yeah. These

8:21

days, especially with large models and, you know,

8:23

somebody wanted to bring other media for a media

8:25

production company. I could see it. I

8:28

mean, again, you're talking this is a device that lives

8:30

between the consumer like

8:33

the NAS for consumers and

8:35

the enterprise, right? This is

8:37

a 15 drive bay system. And

8:39

in the case with the fully loaded one, it comes with a 2018 processor. But

8:42

that kind of makes sense if you're coming from an enterprise design

8:45

standpoint. So it's for very serious folks.

8:48

And an $800 starting entry point

8:50

for just a chassis and the backplane for

8:53

people that are pretty serious seems

8:55

about right when you can put 15 drives in there and you

8:58

can put your own motherboard in there if you want and your own CPU.

9:01

Yeah, I mean, that's that's kind of nice.

9:03

It is a bit expensive, but it sounds really nice for

9:05

people that are serious. It is absolutely

9:08

a really nice bit of kit. It's

9:10

a premium bill quality. And

9:13

I've got no complaints about it. Really. I

9:15

just I just wish that it was five

9:18

or six hundred dollars, not eight hundred.

9:20

Oh, yeah. That would be a totally different equation

9:23

for me because at that price

9:25

point, it becomes the de facto like

9:27

standard unit of deployment for home

9:30

labbers and folks looking to build on raid

9:32

servers and and that kind of thing. They came

9:34

out with the HL-12. You know,

9:37

the HL-12, 12 bays, 500 bucks,

9:39

because I don't really know if it's the slots you're after.

9:42

It's you want something that's quality, right? You're going

9:44

to put this in your home. Craftsmanship. You're going to

9:46

have your home pictures on there. Right. Craftsmanship. I

9:49

think that counts for something. It does, which

9:51

leads me to a little bit of a

9:53

software experience out of the box that I had some issues

9:55

with too, unfortunately. So the image

9:57

they shipped to me had Rocky Linux. the

10:00

SSD and the

10:02

first thing I did was try to log in as

10:04

you can imagine. So first of all I had

10:06

to look for the IPMI password

10:09

which was printed on a tiny little sticker

10:11

on the motherboard. Their documentation unfortunately

10:14

told me it would be printed on a white sticker on the side

10:16

of the case that wasn't present so I had

10:18

to go spelunking on the internet and find

10:20

that Super Micro Motherboards as

10:22

of 2019 put this unique password

10:25

on the motherboard. And I spent

10:27

a while trying to get that

10:30

password in with the correct username of

10:32

admin and then realising that admin had

10:34

to be all caps. It

10:37

was case sensitive so it literally took me half

10:39

an hour to turn it on. That's on me,

10:41

some of it, you know, but some of it's on the documentation

10:44

too. Once I then got the system

10:46

booted up I was presented with

10:48

a GNOME login screen and I was like,

10:50

ooh username 45 drives. So I look in the documentation

10:53

for the HomeLab server which

10:56

is like 100 pages long. I had

10:58

to get to page 61 before I

11:01

found out what that username and password was

11:04

and it was wrong. It told me the username

11:06

was root and the password was correct

11:09

but it was for the user 45 drives

11:11

not the user root so you know some teething

11:13

issues there and nothing major,

11:16

just some small niggles to fix in the documentation.

11:19

Then I loaded up the terminal once

11:21

I got logged in and thought cool right time to update

11:23

packages as you do like a reflex after

11:26

you've used Linux for a long time. Absolutely.

11:28

And I had to do dash dash skip broken

11:31

dash dash fix something like

11:33

I had to pass three different parameters

11:36

to ignore all sorts of shenanigans that was going on

11:38

with DNF in order to fix a

11:40

bunch of out of the box broken packages which was a

11:42

bit disappointing. Hmm that

11:45

is. I'm not you

11:47

know I'm not sure if I'm actually even just thrilled

11:49

with the choice of Rocky. I do again I can see

11:52

it from the enterprise perspective. Well I think

11:54

they ship Rocky because they can do cockpit

11:56

modules and they spent a bunch of time

11:58

on their Houston UI. which is actually

12:01

based on top of cockpit and they've shipped

12:03

some custom modules in there which let

12:05

you see which drive is in which physical slot

12:08

on the system Some

12:10

proper system integrator level stuff, which

12:12

is brilliant. That's exactly what I want to see at this

12:14

price point And then the

12:17

only final Niggle I had

12:19

was I loaded up Firefox because

12:22

it has a desktop for some reason this server comes pre-installed

12:24

with a desktop Whatever, that's fine

12:27

And Firefox had someone else's browser

12:30

history in it Presumably the tech that was setting

12:32

up the server like it was looking at super

12:34

micro fan control and a bunch of other stuff that

12:36

They googled and gone through the github link and downloaded

12:39

the script and I was just thinking

12:41

to myself That

12:43

cheapens it a little bit because I would have expected

12:45

them to have a pre-made Golden

12:48

image with all of that stuff removed and you know,

12:51

I come from the enterprise world with red hat for a long

12:53

time Where building those kinds of experiences

12:56

out of the box experiences is exactly

12:58

what tools like satellite and ansible have been

13:00

doing for? Years and years and years

13:02

and I just I wonder why that wasn't

13:04

done here. It was a custom bespoke

13:07

edition Just for me. I Would

13:11

imagine too like that image is probably gonna be something

13:13

they're developing quite a bit. They'll be probably

13:15

updates their shipping Yeah, but knowing

13:17

you I don't I don't really see you wanting to run rocky in

13:19

production Are you planning to keep rocky install

13:22

on there? Are you gonna try going with something else? I

13:24

want to try putting nix on this bad boy There

13:26

you go. I actually want to do a full-on nix

13:29

OS server One of the guys

13:31

I work with has been showing

13:33

me some really ninja level stuff

13:35

with nix And system d n

13:37

spawn containers and how he's configuring

13:39

that with a like a helper script as part of his

13:42

flake setup I'll get in some details

13:44

into that over the next few weeks months because

13:46

it's it's pretty complicated and I'm still learning

13:48

it myself but it looks amazing

13:50

and Isolated containers

13:52

on the host as part of a nix declarative config

13:55

yada yada yada. Yeah, it's

13:57

pretty cool stuff I'm feeling it. That's what that's why

13:59

I want want to do our server here at the studio as

14:01

Nick's. We almost went Proxmox and then I started

14:04

thinking about it and like, wait a minute, we've missed

14:06

the window Chris, it's mid November. We should have done Nick's

14:08

November. I know,

14:11

I know. Nick's our new year maybe. But

14:14

so, I mean, it really should have no problem, right? It's

14:16

a standard PC, it's an

14:18

X86 box. Yeah. It should just load right

14:20

on there. It's just a Xeon box with a

14:23

CPU and memory. It's

14:26

just a Linux box, right, at the end of the day. I

14:28

know it's early but I mark my

14:31

words, in a couple of years, hardware

14:33

vendors like 45 Drives

14:36

and 45 Homelab and I also

14:38

think iX Systems, they

14:40

will realize that

14:42

to achieve the level

14:44

of quality and reliability they want,

14:47

it will be in their best interest to

14:49

ship this hardware with Nick's OS. I had

14:51

an absolutely fascinating conversation

14:54

with an individual in El Salvador who's shipping hardware

14:56

to stores and by a

14:58

process of elimination, they started with Debian

15:01

and went through everything else and by a process of elimination,

15:03

they're now using Nick's OS because of its quick

15:06

ability to deploy, recover and do all that stuff

15:08

that they need. I think a lot of hardware vendors

15:10

are going to get there. So, you know,

15:13

you'll have to keep us updated on what it's like to run it on this machine.

15:15

Well, I absolutely will. So, I think the way I'm going to split

15:18

my Homelab up now is I'm going to keep

15:20

my existing server with the i5-8500

15:23

CPU in it and Quick Sync. I'm

15:25

going to keep that for all the mundane sort

15:27

of media serving tasks that I have in the house

15:30

and then this one is going to be straight

15:33

up like business suit and tie, you

15:35

know, photo storage, next

15:37

cloud, that kind of stuff that, you

15:40

know, I'm going to throw a few hard drives in there

15:42

and do ZFS, all that kind of stuff. So, I think

15:46

for me it'd be nice to have that separation of duties

15:48

as well between like a serious server and

15:50

a fun server. Do I need it? No,

15:53

but it is fun as hell. Alex,

15:55

I know often when you're reviewing these

15:58

kind of units, energy... consumption

16:00

comes to mind for you. Have you measured this yet? Now

16:03

whilst I don't have 15 hard drives to load

16:05

this thing up with what I can tell you

16:07

is what the Standby power consumption

16:10

is because this thing has a BMC built into it for

16:12

proper IPMI because it's a super microsurfer

16:14

motherboard So just turned

16:17

off plugged in with just the BMC

16:19

on is 23 watts Which

16:21

if you consider that is already more than most

16:24

of those small form factor like 1 liter

16:26

PCs on its own True

16:28

gives you an idea of things but

16:30

better than those Dells that

16:33

we have plugged in Yeah,

16:35

you can't throw two terabytes of RAM

16:37

into a micro PC Can you I mean the more we talk

16:39

about it? I really see how this fits between like a one liter

16:42

PC home lab machine and also

16:44

like a huge used like super micro

16:47

Or old Dell or HP rack unit just

16:49

powered at rocky Linux just at the desktop

16:52

No hard drives installed nothing like that

16:55

roughly roughly 120 watts It's

16:58

a Xeon at the end of the day and that has Some

17:02

ramifications with with energy usage. So if you

17:04

compare that to like the i5 8500 for example, that's drawing 10 to 15

17:09

watts for idle. Okay, you've got a bunch of hard drives

17:12

in that box too But that's again,

17:14

we're be really nice to have click sync in that CPU

17:16

so you can throw a few more tasks out Yeah,

17:19

now it's a server. So they've put server grade

17:21

fans in it The CPU itself

17:24

doesn't have any fans on the heatsink So

17:26

that should give you an idea of the level of noise that

17:28

you can expect from the other fans in the

17:30

chassis, too You could legitimately

17:33

condition the air in your house with

17:36

this thing If you put a HEPA filter on the front

17:38

of it, the airflow is quite impressive So a

17:40

big thanks to 45 drives for sending that unit

17:42

over for review and you'll be hearing more about that in the

17:44

upcoming episodes That's

17:46

a great way to support the show you get a hundred

17:49

devices and you can see that tail scale truly is simple and secure

18:00

but we love that it's built on WireGuard. It'll

18:02

get up and running in just minutes. TailScale's

18:04

so great when you're traveling. I

18:07

mean, it's fantastic if you just work from home and

18:09

you wanna have a secure connection

18:11

to maybe a VPS and something that's running on your local

18:13

machine and a phone. You can kind

18:15

of create your own mesh network across all those

18:18

different networks. That's really handy. But

18:20

man, oh man, when I'm traveling,

18:22

oh boy is it great. Knowing

18:25

that every photo I was taking, every contact

18:27

I was saving, everything was being synced securely

18:30

over TailScale to my next cloud box, that's

18:33

a level of peace of mind that I just haven't

18:35

really been able to describe before when I'm joining like an

18:38

airport Wi-Fi in El Salvador. I

18:40

just love knowing that all my information's

18:43

protected by TailScale. And TailScale

18:45

Traverse Double NAT like it's nothing.

18:48

And when I'm switching between networks, going between

18:50

weird cellular providers I've never heard of and

18:52

onto an Airbnb Wi-Fi, TailScale

18:56

never missed a beat. My connections

18:58

between my servers, my computers, my mobile

19:00

devices, even some of my

19:02

appliances was just bulletproof. I

19:04

was managing my power input on my

19:07

Victron system using a subnet

19:09

router on my LAN and my RV connected

19:11

over cellular. And I'm doing

19:14

it all from my phone while I'm traveling.

19:17

It really is a great resource too for moving data

19:19

back and forth. They have TailScale Send which made it really

19:21

easy to copy files back and forth. If you wanna

19:23

share resources with folks, you can either integrate

19:25

it with your enterprise ACLs or

19:27

if you're just an individual, some friends, they got a system that

19:30

works for you. It'll completely

19:32

work when you're separated by firewalls and subnets.

19:35

It's just, it really is the chef's

19:37

kiss of networking. It's where I wanted to

19:39

see WireGuard go when I first heard it get described

19:41

way back in the day when it was just a twinkle

19:43

in the colonel's eye. From arch servers

19:46

to mobile devices, you've gotta try it out, deploy

19:48

it, and see how fantastic it is to build your

19:50

own private mesh network. Just

19:52

go to tailscale.com slash self-hosted.

19:55

You can try it for free for up to 100 devices, really

19:57

make sure it works for you, and it's a great way to

19:59

support the show. So it's tailscale.com

20:02

slash

20:02

self hosted

20:05

Well two weeks ago. We talked about the verge cast

20:07

and their big self hosting Podcast

20:09

episode and in that they interviewed

20:12

Alex who is the lead developer for the image

20:14

project now image is Ostensibly

20:17

a Google Photos self hosted

20:19

replacement or at least that's what it's trying to be So

20:21

Chris and I sat down with him to talk through

20:24

the future of the project and the roadmap and also,

20:26

you know Open source sustainability and that

20:28

kind of thing So we

20:30

finally got Alex on the show the developer

20:32

of image and it's kind of great timing

20:35

because version 1.8 6 just

20:37

came out and Alex welcome

20:39

to self hosted and sorry we haven't had you on the

20:41

show before now. No worry Thank you for

20:44

having me is the pleasure to be here to talk

20:46

to you guys and to talk about image Well,

20:48

the new release came out just a couple of days ago as

20:50

we record. It's got some nice features

20:53

I think specifically like around sharing right that

20:55

is correct Yeah, so

20:57

this new release we add the unified

21:01

Sharing for partners. So

21:03

now if you have multiple partners, you

21:05

can enable that option so all

21:07

of the access of the partners always is

21:09

also showing on your timeline and This

21:12

make browsing Experience

21:15

a lot better so you don't have to switch to

21:17

the sharing tab and then click on each partner

21:20

to see the new Media

21:22

but now just in your timeline But

21:24

there are a lot of caveat as this is the

21:26

basic version. The searching is

21:29

not including the partner media

21:31

yet

21:32

and

21:33

The only thing that share on that

21:36

media is the the location

21:39

For now, that's fantastic. Because if you

21:41

think about how most people actually use a photo

21:44

solution, right? My wife is

21:46

taking pictures of my kid I'm taking

21:48

pictures of my kid and we probably both want

21:50

to see pictures of our kid, right?

21:53

And we say it's a family unit thing and

21:55

a lot of the self hosted solutions

21:57

to date don't really take

21:59

that much multi-user within a

22:01

single household, a single family into

22:04

account. So I'm absolutely thrilled to hear that you guys

22:06

are. Thank you. And I think this

22:08

is coming from a lot of my

22:11

own use case as well, because scratch

22:13

the own, scratch your own itch. That's the best way to

22:16

develop software, in my opinion. Yeah. And then we

22:18

add the new map tire server to moving

22:23

away from using the free tire

22:25

server from OpenStreetMap as

22:27

the usage policy. I

22:30

thought that was really funny when I looked at it, because

22:33

you were like, well, when we were just a small scrappy

22:35

project, we could get away with using it for

22:37

free. But now we're kind of a big deal. So

22:40

we kind of need to do this properly by the book.

22:42

Yeah. The community, there

22:44

are some concerns that

22:47

asking us that we should move away as soon

22:49

as possible to not

22:51

overuse their free

22:53

resources. And we take

22:56

this change and we move away to a

22:58

different map tire server and

23:02

it's also giving us the opportunity to build

23:04

our own layers. So to

23:06

make it look slicker on the

23:09

dark and the light mode.

23:11

I think it does. I think it does.

23:14

So I want to talk about the audacity

23:16

of the project for just a moment, because the idea

23:18

of trying to replace Google Photos kind of seems like

23:21

an impossible task. And yet version

23:23

one dot eight six getting pretty

23:25

far now with face recognition, sharing

23:28

support, the location.

23:30

When I was looking at my pictures from my trip, the

23:32

new map stuff is really slick. So you

23:35

are kind of in a transition phase now to

23:37

a big project, right?

23:39

Yeah, I would say so. Just

23:42

to look at this core

23:44

member,

23:45

because they kind of group the

23:47

new user coming in by

23:49

day. So when you look at it, you can

23:52

see it grow. And

23:54

when I first started this core, we might

23:57

have one or two new people

23:59

come coming in. the server every

24:01

day and now when you look at it, it

24:03

can reach like 20-40 people. And

24:07

then the star

24:09

on GitHub is also the indication

24:12

of how the project is getting more popular.

24:15

And also, more

24:18

often than I would admit that I would browse

24:20

the internet to see how people are talking

24:22

about image. And yeah, we've

24:25

seen a lot of chatting and

24:28

most of the girls are coming from a worth

24:31

of miles. Yeah, it's coming up online all

24:33

the time now though. I feel like I see it getting mentioned

24:35

every week at least by some different group online

24:38

saying they're using it to replace Google Photos. People

24:41

are recognizing it for a genuine

24:44

replacement. It's pretty exciting. I

24:46

just wanted to say congratulations.

24:47

Thank you. My goal is

24:50

to build something for myself and

24:53

find image maybe what

24:56

it is. You

24:58

and I have crossed paths a couple of times on Reddit

25:00

I think in the past and talked about your funding model

25:02

in particular in terms of a

25:05

sustainable model for an open source project.

25:07

I mean, we've seen Home Assistant with Nebuchadse for

25:09

example build a company around

25:12

an open source project.

25:14

I think

25:15

we've also discussed in cross paths that

25:17

something like this photo

25:20

solution is so big

25:22

in scope that you do almost need a company

25:24

to support it. You've obviously

25:26

done amazing work and you've got some fantastic contributors

25:29

to date but I wanted to ask you what the plans

25:32

are for the future of image as a

25:34

project and also as a sustainable project

25:36

in open source. So I think there are

25:39

one thing that I

25:41

want to not never pivot

25:44

off is to not making it into

25:46

some type of paywall like

25:49

some of the open source projects

25:51

starting out as open source and then they got funding

25:53

from outside

25:55

investor and then they got that money pressure

25:58

to eventually have to make money. of

26:00

pivot of the first original idea.

26:04

So I don't want

26:06

to do that for image at all. I want to make it as

26:08

something good, something people

26:11

enjoy using. So that is something

26:13

that we would never like to move away

26:15

from. The car team

26:17

has talked about maybe eventually after

26:20

the service, the app is more stable.

26:23

We could think of building a

26:27

hosting solution for other people. So

26:29

people that would like to use image but don't

26:31

want to host it themselves. So

26:35

I think that is the, this might

26:37

be the only way right now to

26:40

make it as a sustainable

26:43

business. But by not

26:45

making it as the main

26:48

income source at home, for

26:51

all of us, I think that's making

26:53

it as a sustainable project

26:55

that keeps growing. Because we don't

26:57

have to depend on it for our livelihood.

27:00

You still have your day job and you still

27:03

dedicate your time outside

27:05

of job in order to

27:07

build image. And it's a

27:11

joy to do that as something,

27:14

I think the whole core team really enjoy

27:16

because we have such a nice group to

27:19

chat about technology, code,

27:22

and just how to

27:25

beat Google. You

27:27

love to hear that, right? And

27:29

really, it's great that while you grow

27:32

the project and get it to that point where it's stable

27:34

enough that it could be a product, that you do have the

27:36

support and income of a day job. And

27:38

maybe one day those two things merge and

27:41

image becomes a bigger part of your income source.

27:43

Yeah, and that's what I

27:46

try to achieve is when

27:48

it gets to that point where I can

27:50

replace the income source of image

27:52

into my own, that

27:55

I can support my family and the growth of

27:57

my family, then that would be fantastic.

28:00

work on something you love, you make good

28:02

things for people, and then you also get to earn money

28:04

from it. Yes, that would be absolutely fantastic.

28:08

How could people support the project today?

28:11

Today, there is only donation

28:14

means to support the project besides

28:16

testing and reporting bugs to

28:19

help us develop the project

28:21

better. You can support the project

28:23

financially from GitHub sponsor

28:26

or from Leapro

28:29

Bay Pay or buy

28:32

me a coffee. I drink a lot of

28:34

coffee, so keep the coffee

28:36

coming. In

28:39

the Vergecast interview that we talked

28:41

about on the last Self-hosted episode, you're

28:44

now a super famous Big Deal developer

28:47

and the big time on the Vergecast podcast.

28:49

I wouldn't say so. You

28:52

talked a little bit about your architectural

28:54

decisions about splitting it out into microservices

28:57

and a bunch of Docker containers and stuff like that. Is

29:00

there anything you wish you'd done differently now

29:03

or are there any big changes coming down the

29:05

pike? Yes, so the goal is

29:07

to

29:09

consolidate some of the services

29:11

like the proxy container and

29:14

the web container to put that into the

29:16

server container to make it less

29:18

scary when people are looking at the Docker Compose

29:21

file because those can be done. But

29:25

when I started it, it makes sense

29:27

to split it up at that moment.

29:30

And now if we might have found a way

29:33

just today to make the

29:35

web not server-side

29:37

rendering in anymore while still

29:40

keeping that really snappiness when

29:43

using the app.

29:44

So with microservice architectures,

29:46

you have the potential for remote

29:49

workers and stuff like that. Correct. I

29:51

mean, I would love to see it where I could distribute the facial

29:54

recognition. So I just dumped, for

29:56

example, like three or four terabytes worth of photos

29:59

into image. in the last couple of months because

30:01

you guys are awesome and added external library

30:03

support, which is amazing. Mwah, chef's

30:06

kiss. Thank you. But it

30:08

took like two or three weeks for it

30:10

to chew through all of those images

30:12

that I had. I've got lots of CPUs

30:14

in this house. It would have been great if I had an easy

30:16

way to distribute that task throughout

30:19

all my systems. Yeah. So

30:21

actually the idea of

30:23

breaking down microservices,

30:26

different containers, handle different things

30:28

is exactly to serve this use case. So

30:31

as of now, you can run

30:33

the machine learning on

30:35

completely different machine. And if

30:37

you put a frontend of load balancing

30:39

on it, then you can distribute it on

30:41

multiple nodes because we are

30:44

not accessing the file

30:46

on that container, but you

30:48

are sending the file

30:51

to those containers and then it would do

30:53

the referencing and then sending back the data

30:55

that it needs. The microservice

30:58

is a little bit more involved

31:01

because you would need to be

31:03

able to see the file system similar

31:06

to the server so that it can access and

31:08

do the transcoding stuff. But if you

31:11

have network mounted, you can also do

31:13

that already. Pretty typical distributed

31:15

network storage replication

31:17

problem there. Yeah.

31:20

But I think the

31:22

architecture has been designed

31:25

exactly for that

31:26

purpose. And we have a lot of people

31:28

that already do that. I think that will help you

31:30

guys as you move towards eventually

31:33

a hosted service. You will be able to pick the

31:36

cheapest VPS provider that week. Yeah. Do

31:39

you know it maybe? Yeah, we got a great

31:41

coupon code you can use. Well

31:44

I think for me and probably a lot of people

31:46

listening, I know Alex too, this is one of the tools

31:49

that has really been

31:51

the key to de-googling so

31:53

much of my private information. And

31:56

so privacy and security is super

31:58

important. I'm just curious to know your thoughts. around

32:01

enabling sharing but keeping people's

32:04

images safe and secure and how that kind of

32:06

works from a back-end perspective. Is your

32:08

question more about at

32:10

home? Yeah, like say if I wanted to share an image

32:12

with somebody, would there

32:15

maybe be like a back-end service like down

32:17

the road, some sort of intermediary that would be

32:20

the go-between to share the image so I wouldn't have to expose

32:22

say like my home machine or something? I'm

32:24

trying to think of ways of sharing images where I don't

32:27

I don't link someone to my image instance. That's

32:30

so challenging at the moment

32:32

but anything might

32:35

have a solution. We're

32:36

on video chat with Alex, dear listeners and

32:38

I can see the cogs turning right

32:41

now live as you're thinking what

32:43

if we had some kind of like remote cloud

32:46

you know. Yeah, that's what I'm wondering. VPS

32:48

like linked up or something you know that's

32:51

great idea Chris. Feature requests right

32:53

there. Turkey Days coming in and I'm thinking something

32:56

that I could like a page that would be generated

32:58

that I could share with family but I wouldn't have to link

33:00

them

33:01

they wouldn't all be necessarily loading it from my direct server.

33:04

I see so in that page you would talk

33:06

to the image server at home but that is

33:09

abstracted in the background and then

33:11

it would use the image API

33:13

in order to pull those just like a proxy

33:16

right just like a proxy pay. Eventually

33:18

this is something that we want is to build a plugin

33:21

system for the

33:23

people to build things around

33:25

an image. The API

33:28

is compatible with a lot

33:30

of the languages so we used open

33:33

API so it can generate

33:36

SDK for Python for JavaScript

33:39

for Rust if you want

33:41

for anything that you could think of right and

33:44

then they can use that in order to talk

33:46

to their server the image instant

33:49

and now one of the we have one really

33:51

good tool I would like to mention here is called image

33:54

go is from Simulat

33:56

it has this court hand

33:58

handle

33:59

This is a CLI

34:02

tool. You point it to your Google

34:05

Tech Al. You don't even have to extract

34:07

all the zip file that you download. It

34:10

unzip it and then it pars all

34:12

the metadata from that JSON file

34:14

that Google gave you. And

34:17

then upload everything into image

34:20

and create the album that comes

34:22

in your Google account. So it's just like one

34:25

click go. That's

34:27

so neat.

34:28

Yes. I love that. See, that

34:31

plugin system, if that really extends, it could be

34:33

a solution because people could build things around that. Because

34:35

when I really A-B, when I miss from Google Photos,

34:37

it's like a way to share photos and I'm not

34:40

like sharing any of my personal infra. But

34:42

something down the road, like a plugin

34:44

system, there's all kinds of ways to solve that. I'm

34:48

thrilled to hear that you guys are thinking more about that too. It's

34:50

going to be an exciting future as an image

34:52

user. Absolutely is. Now,

34:54

I have another question for you, Alex. Ever

34:57

since the beginning of the project, there's

34:59

been a banner in your documentation

35:01

and on your website that says, here be dragons.

35:03

Do not do anything serious

35:06

with this project because it's still under very active

35:08

development. Do you have any senses

35:10

or time scale as to how long before we can

35:12

actually start to air quotes

35:15

depend

35:16

on image? You know, in this court, we

35:18

have a funny way of saying that this is the any

35:21

question is a

35:23

prohibit question. We don't

35:25

talk about that here. Yeah,

35:28

fair enough. Don't commit to deadlines on air

35:30

in particular. I get it.

35:32

We put that because part

35:34

of it, this image is very active in

35:36

development, right? It's like just this

35:39

last release, the first time ever

35:41

we pushed out the first

35:43

release that if you don't, if you're not on

35:46

the same version, you cannot access

35:48

the instant from your phone at

35:50

all

35:51

because we changed the underground,

35:54

the underlying API. That's

35:57

the only way to resolve the difference is

35:59

to lock out. and lock back in on the same version.

36:02

Things like this could happen because we

36:04

are striving to go to the

36:07

industry standard for the

36:09

code base. If anything that we see

36:11

can be improved for the better future

36:13

of image, we will do

36:16

that. And since there are

36:18

many things that we still can improve,

36:21

we don't want to prohibit us to

36:23

backward compatibility

36:26

at the moment. But people

36:28

are striving to get to the point of feature

36:33

freeze or when the core

36:35

architecture is more

36:38

stable, we

36:41

would like to eventually

36:43

remove that

36:44

banner.

36:45

I think we all just see the potential of the

36:47

product and want to start using it. I

36:49

say product. It's not a product. Project.

36:51

And we just want to start using it today because

36:54

I'm real fed up of the

36:56

cloud nonsense and giving our machine

36:59

learning, overlord, model, so

37:02

much data. So I think that's

37:04

the only pressure really from us is that we kind

37:06

of wish it already existed because the pressures

37:09

that forced you to create it in the first place

37:11

are quite powerful. Yeah, very tempting.

37:14

They're tempting fruits, aren't they? Alex,

37:16

I'm going to keep using it even with that banner on there.

37:18

So there. And thank you so

37:20

much for creating it. And thank you to the entire team and everybody

37:23

that works on it and contributes. Absolutely. I

37:25

know a lot of us in the self-hosted community are just

37:27

extremely grateful. Yeah, I would like to shout out to the team

37:29

too. Everybody in the team

37:32

is great. They come from different backgrounds.

37:34

We have students from Europe. We have people

37:37

working in the application security, some

37:40

11x developer.

37:42

So the whole team is great.

37:44

And people are very

37:47

welcoming and willing to teach

37:49

each other. So I want

37:52

to keep this mindset growing in order

37:54

to attract more contributors

37:57

and just to make it a good place to be.

37:59

not just from using the application, but from developing

38:02

the application as well. And so if you'd like to contribute,

38:05

you can go to github.com and image-app,

38:07

I believe, is the place to go. You've

38:09

got a couple of hundred contributors already. A few more

38:11

wouldn't hurt, right? Yep.

38:15

So a big thank you for me and Chris for joining us on

38:17

the show today. And I'm sure this won't be the last

38:19

time that you check in with us. But until

38:22

next time,

38:22

thank you very much. Thank you guys. Thanks, Alex. Thanks,

38:25

Chris. You guys have a wonderful night.

38:28

So, I'm Chris

38:30

H. Linode is where we host everything in

38:32

the cloud. And now Linode is part of

38:34

Akamai, the Akamai. But with all the

38:37

tooling, the UI that we

38:39

like, the CLI that's super handy, all

38:41

that stuff that you can use to build, deploy, and

38:43

scale in the cloud, that's

38:45

still there. It's just now they're combined

38:48

with the power of Akamai's global reach.

38:50

They are the premier network out there. And

38:53

Akamai is expanding

38:55

their cloud services, expanding

38:57

tooling, but keeping things affordable,

39:00

reliable, and scalable

39:01

for users of any size, an enterprise

39:04

or an individual project. And something we've taken

39:06

advantage of in the last month is the expansion of the data

39:09

centers. They're expanding worldwide to give us more

39:11

access to more resources so you can grow

39:13

your project or your business. So

39:15

why wait? Go see why Linode is the only cloud

39:17

provider we recommend. Experience the power

39:20

of Linode, now Akamai. Go

39:22

to linode.com slash SSH. That supports

39:24

the show, and it gives you a $100 credit

39:26

so you can really kick the tires. So

39:28

go see how Linode, now Akamai, can help scale

39:30

your applications from the cloud to the edge.

39:32

That's linode.com slash SSH.

39:37

Well, at that time of year, Brent, you messaged me saying,

39:39

I want to buy some hard drives, and I couldn't

39:41

refuse the opportunity to sit down with you and

39:43

discuss hard drives. It's literally

39:46

my specialist favorite subject in the whole world.

39:49

Once a year, you get all giddy about getting new

39:51

hard drives, and I thought I'd join in this year. So here's

39:54

part one. Brent and I sat down and recorded

39:56

a few different segments to use over

39:59

the holiday season. and discussing

40:01

his journey into buying hard drives

40:03

and purchasing considerations, all that kind of

40:05

stuff. I'm going to put part two

40:07

in today's post show, which I believe, Chris,

40:10

we have a special offer for our members. It

40:13

is the Black Friday season after all. So

40:15

if you've been thinking about becoming a member and support the show and

40:17

you want to get that post show, use the promo

40:19

code Black Friday and you'll get $2 off a month for

40:23

a whole year of the SRE membership.

40:26

Just go to selfhosted.show slash SRE

40:28

and it's promo code Black Friday, all

40:30

one word. And then you get the post show and you get a

40:32

post show every single episode. So

40:36

all the way back in 2019, October 2019, four years ago, Brent,

40:39

you and I sat down with

40:44

Antonio, the developer for Merger

40:46

FS, and you professed to me in that

40:48

episode and public record,

40:51

no less, that you wanted to take

40:53

storage more seriously. And here we are four years

40:55

later and I think it's

40:57

about time we cashed that check. It's

41:00

long overdue, Alex. And I admit it

41:03

had been on my mind for four years previous to that

41:05

as well. So I am

41:08

in bad shape because we were talking about this before

41:10

we pressed record. And you said

41:12

to me, I've been buying drives with ZFS in mind

41:14

in pairs for like the last eight

41:17

years. No more than that.

41:20

Since my photography career required

41:23

it, which is, I had

41:25

been doing photography for 18, so I've got

41:28

paired one terabyte

41:30

drives and paired 500 gig

41:32

backup drives. Basically anytime

41:35

I'd buy drives, I would try to pair them because

41:37

I knew one day I would be smart enough to actually

41:39

do something about it.

41:41

And that day is today, is it?

41:43

I'm hoping it is, but my

41:45

level of success so far is challenging

41:48

at best. Well, okay. Let's

41:51

start at the beginning. You mentioned photography.

41:53

Now, I know that's a career path that was

41:56

huge for you previously, but now obviously

41:58

with your next cloud work. I would imagine

42:00

your data generation is a little,

42:03

you know, the velocity is a little slower on that side

42:05

of things. What kinds of stuff

42:07

are you looking to store at the moment?

42:10

Yeah, it's a good question. I think

42:13

most of my data, at least the

42:15

big stuff at this point,

42:17

is archival, to be honest. Like

42:19

it can just sit there and

42:22

won't get access very often, then it's a read-only

42:24

probably kind of scenario if ever.

42:26

Okay, so that leads me into my next question. If

42:29

it's archival, do you

42:31

genuinely need it to be always

42:33

on? Always accessible.

42:36

Hmm. I hadn't even considered that because

42:38

I had just pictured all of my storage being

42:40

in one box. But tell me

42:42

what you have in mind. If it's an archival situation,

42:45

it's not beyond the realms of possibility

42:47

that you've got a client ringing you up that you had

42:50

on the books 10 years ago that says, you know that shoot

42:52

you did in our mine, could you

42:54

find that particular, you know, shot

42:57

of Fred or whatever? I've

43:00

gotten those emails before. Yeah, I'm sure you have. And you think,

43:02

right, well if I just, you know, write down

43:04

what's on each hard drive, external hard drive

43:06

in a drawer, for example, that

43:08

could be a legitimate filing system for

43:10

that data. The issue

43:13

with that is Bitrot is a thing

43:16

and that keeping those drives just sat in a

43:18

hard drive, you know, they're not going to last forever.

43:22

So you could explore something like

43:24

Amazon's Glacier service,

43:26

which is a cloud based long term archival

43:29

situation. It's fairly cheap.

43:32

They do kind of hit you on the retrieval fees.

43:34

But if this is an archival situation only,

43:38

maybe it's a decent option. I hadn't

43:40

considered that at all as my primary

43:42

source of storage. I had only considered

43:45

cloud solutions as kind of a offsite

43:47

copy. I think mostly

43:50

because I have, I don't know

43:52

if I

43:53

trust them with the primary data.

43:56

Is that one way to look at it? But you're suggesting maybe

43:58

I have drives in a drawer and also... so

44:00

some data there, is that what you're suggesting? It

44:03

could be. I mean, the other option is you

44:05

just have all of that data running all the time

44:07

in your house, and it's always available on

44:09

top of ZFS so that you've got all the checksumming

44:12

and beautiful magic that ZFS gives you for data

44:14

integrity. And then you back

44:16

that up to Glacier. And the purpose for suggesting

44:19

that is it would reduce the

44:21

complexity for you somewhat in terms of having

44:23

to have a box at a remote house with, as

44:26

I found out fairly recently, having

44:29

a remote box in a different country or a

44:31

different side of the country even, even

44:33

if it's in a family house, it's

44:36

hard sometimes to get access to that. And the

44:40

economics of something like rsync.net

44:42

don't really work out. Seven

44:44

or eight terabytes in rsync.net is 70

44:47

or $80 a month. Oh,

44:49

I see what you're saying. Which pays

44:52

for a remote backup server in no

44:54

time at all. But Glacier

44:57

is, from memory, it's 10 bucks

45:00

for a similar thing or less

45:02

per month. Like I say, the

45:04

way they get you is on the retrieval. So if you try and do

45:06

an expedited retrieval, they'll charge

45:09

you $10 for a thousand API requests versus

45:13

just a standard rate of 3 cents per 1000 requests. Hence

45:18

the name Glacier. Because it's like,

45:21

only if the world is going wrong do you want to melt that thing.

45:24

Yeah, they've never actually published what the underlying

45:26

storage technology is. I suspect it's tape,

45:29

but we don't actually know. And so it would

45:31

make sense because I would imagine they have

45:33

those tapes in some kind of robot

45:36

automation system that can actually move

45:39

a physical piece of media in and out of

45:41

a specific location. And

45:43

if you're doing an expedited recovery, that probably

45:45

means some S hit the fan somewhere. And

45:48

you need that stuff back pretty quick. All

45:51

right, so let's leave Glacier for a second

45:53

and talk a little bit more about your local

45:55

hardware.

45:56

Desires, you sent

45:58

me a list of all the hard drives. you've got and let me just

46:01

let me just read this to the audience because it's

46:03

it's quite glorious I think you're going to enjoy this

46:05

is it really yeah I love I liked

46:07

it there is a Western Digital 8 terabyte

46:10

drive brackets shucked a

46:13

Western Digital red 4 terabyte drive

46:15

a Western Digital caviar black 2 terabyte

46:18

drive 3 seagate 1.5

46:21

terabyte drives 2 Western

46:23

Digital caviar 1 terabyte drives

46:26

and 2 500 gig drives

46:28

how many is that one two three four five

46:30

six seven eight nine ten ten drives

46:33

Brent that's quite a lot

46:35

unfortunately this doesn't include any of

46:37

the two and a half inch like external drives

46:39

that I've been carrying on all my travels to

46:42

do you know local backups as I go so

46:44

there are a couple more of those maybe we should throw

46:46

in the mix but they might not be useful in the setup

46:48

I don't know and then the next line in

46:51

in our little chat exchange was

46:52

but

46:53

the data duplication inefficiencies

46:56

galore etc so really

46:59

I have no idea quite how much storage I

47:01

have so if we add all

47:03

of that storage up dear listeners it comes

47:05

to approximately 21 and a half

47:07

terabytes but your guesstimation

47:11

on the use of that is somewhere

47:13

between 8 and 13 terabytes

47:16

is that right

47:17

yeah I think part of the challenge

47:19

here is that these

47:22

drives were never all available at the same

47:24

time and so they ended up being

47:27

generations of storage

47:30

for me you know someone fill up and then

47:32

they would go on let's call it long-term

47:34

storage and be replaced and

47:37

as they filled up depending on how

47:40

busy I was at the time or what it would just kind

47:42

of get slotted in and so the data is not

47:46

extremely well organized you know it's organized as

47:48

well as I could muster along

47:50

the way but because of that drive

47:52

swapping

47:53

there's also duplicates of stuff especially

47:56

with system backups and things like that there

47:58

are multiple multiple copies of that

48:00

kind of data and photography

48:02

projects. Sometimes on one set of

48:04

drives there's a project that was half finished

48:07

and on the next set of drives there's the whole project that

48:09

was finished or something like that. I

48:11

think if we approached

48:13

this intelligently and got rid of

48:16

all those duplicates, I would imagine there's way

48:18

less data than I think there is. But

48:20

to be honest, I'm just human and I have

48:22

no idea. You know what's interesting, I

48:24

think we've all been there. We've all found an old

48:26

hard drive in a drawer and gone, hmm, you know

48:28

what? I should back this up and then a couple

48:30

of years, elapse, and then

48:32

you find that same hard drive in a drawer and you go, you

48:35

know what? I should back this hard drive up. And

48:37

before you know it, you have that hard drive

48:39

in triplicate somewhere named slightly

48:41

differently, you know, every time. Well,

48:44

the thing for me is like,

48:46

this is just another iteration of my trying

48:49

to get that great solution. I mean, you've got

48:51

perfect media in the server, right? And

48:54

I've had various versions of

48:56

tons of hard drives in a box

48:59

and various versions of having it be

49:01

functional for me, but it never quite

49:03

got to the point where I felt really

49:06

comfortable about the data and that

49:08

it was, you know, being checked for parity and

49:10

that if a drive failed, that it wouldn't lose anything,

49:13

etc., etc. So I

49:15

still, despite my,

49:16

you know, knowing better and my efforts, I have

49:18

a long way to go, I feel like, which

49:21

is embarrassing, to be honest. Well, in

49:23

that case, my suggestion

49:25

is keep it simple.

49:28

Do a pair of disks

49:30

in a ZFS mirror and

49:32

then never think about it again. I mean, obviously monitor

49:35

them and all that kind of stuff. What I mean in terms of

49:37

you worried about like check summing

49:39

and all that kind of stuff and parity and then data

49:42

integrity, ZFS gives you all

49:44

of that out of the box because it doesn't

49:47

actually care about specific files.

49:50

ZFS cares about the blocks, the data

49:52

blocks, and it checks that if a specific

49:55

block is read from disk and doesn't

49:57

match the check sum it expects between the two mirrors.

50:00

it will go away and figure out what the right one is. And

50:02

then if it can't give

50:04

you the fully integral

50:06

data, it just

50:09

won't give you that piece of data at all. Which

50:11

sounds like a bug, but actually if you

50:14

ask me, it's a feature because it means that

50:16

you're never gonna get garbage out of a ZFS system.

50:20

The other nice thing about doing a pair of drives

50:23

like that is your unit of expansion

50:25

becomes really simple. And these

50:28

days you can buy 20 terabyte

50:30

hard drives for $300 each, which

50:32

isn't cheap granted, but

50:36

also for 20 terabytes of mirrored

50:39

storage, effectively 40 terabytes

50:41

of raw, 20 terabytes of usable.

50:44

That's actually not a bad deal at all. The

50:46

other thing about doing a pair is slot

50:49

density matters. You

50:52

might have, let's say four slots

50:54

in a server. So

50:57

let's put Brent in the shoes of five

50:59

years in the future Brent and this 20 terabyte mirror's

51:01

full. You think to yourself, well,

51:03

what can I do here? Well, you could just go and

51:05

buy another pair of drives of

51:08

whatever the in-vogue size is

51:10

that year. Let's assume it's 40 terabytes

51:12

in five years time. And

51:14

then suddenly you've more than doubled

51:17

your capacity, but all the old data is still

51:19

there in its old format for

51:22

you to move around between as

51:24

you see fit. The mental model,

51:26

it keeps it really simple to understand.

51:29

You know, that leads me straight into the

51:31

whole crux of the question that I have

51:34

today,

51:35

which feels like a bit of emergency. We're at

51:37

Black Friday season and a

51:40

bunch of drives are on sale and if you look at

51:42

them in Canadian dollars, I

51:44

have a potential to save like $200 per drive or

51:47

something with some of these sales. So

51:49

my question was, you know, you're suggesting

51:52

these, get the massive most drive

51:54

that I could possibly muster in my budget

51:56

and throw those as a pair

51:58

into the storage.

51:59

track that I have here. But it

52:02

gets me thinking about all these drives that I already have.

52:04

I've got an 8 terabyte, for instance. Can

52:07

I just buy an 8 terabyte drive and combine

52:09

it with that?

52:11

It seems like maybe

52:13

that would be far cheaper, and maybe that would

52:15

get me far enough where I could wait a year or two

52:17

and then get that massive pair

52:19

later. What do you think about the now versus

52:22

later question? The same logic

52:24

applies. Yeah. Get the biggest

52:26

hard drive your budget will

52:28

permit to pair with the old one.

52:31

Even if all you buy today is one 20

52:33

terabyte hard drive, for example,

52:36

pair that with the 8 that you've already got,

52:38

the 8 terabyte. Your mirror would

52:41

be an 8 and a 20. You'd be

52:43

wasting 12 terabytes today, which sounds

52:45

a bit stupid. But in

52:47

six months' time, when you come across another good

52:49

deal, and this just allows you

52:51

to do dollar cost averaging of hard drives,

52:53

I guess. You're buying

52:55

a hard drive now to make

52:58

the mirror and then in six

53:00

months' time, all you do is swap out the 8 terabyte

53:03

for the new 20, for example,

53:05

re-silver it, and suddenly then you've got the full 20

53:08

available to you. I had never

53:11

even considered wasting 12 terabytes

53:14

just for a couple months. It's a bit of a first

53:16

world opulent problem, isn't it? But

53:18

when you put it that way, when you're trying to

53:20

balance budget and maybe drive slots

53:23

and things like that, it

53:24

actually, in a strange way,

53:26

makes a lot of sense. But

53:30

it doesn't

53:31

make it easy to make a choice, does

53:34

it? No. The biggest knock

53:36

against ZFS really, versus something

53:39

like MergerFS, which you know I'm a great proponent

53:41

of,

53:42

is that

53:44

with ZFS you have to start with empty drives.

53:47

My question to you would be this 8 terabyte

53:50

drive, do you have somewhere to put that 8

53:53

terabyte drive whilst you build

53:55

the new array, the new

53:57

Z-pool?

53:59

It's a very... good point. I

54:01

will admit this 8 terabyte I also

54:03

may have

54:04

broke the connector on it so it's a little... Yeah,

54:06

it's easily done. So here's

54:10

the other thing right, we were talking about the data

54:12

you wanted to store and it's all kind of suit-and-tie

54:14

business-y data right? It's it's real

54:17

data that that cannot be replaced

54:19

stuff.

54:20

Yes.

54:22

Why not use your existing fleet

54:24

of mismatched hard drives to store

54:27

some slightly more replaceable stuff instead

54:29

using something like merger FS. Don't worry

54:32

about the parity on that and then have two

54:34

different tiers of data stores within the

54:36

same box using merger FS

54:38

and and that for one

54:40

side for the media and then ZFS for the suit-and-tie

54:43

data on the other.

54:45

I also hadn't considered that I'm

54:47

glad you're here with me. Yeah

54:50

and I like the idea of the peace

54:52

of mind of new drives because these drives you

54:55

know have

54:56

been around and just been sitting there as well

54:58

and they've traveled across the country in the

55:00

back of a trailer etc so my

55:03

confidence in these older drives

55:06

is also surprisingly low so the

55:09

idea of buying new drives feels I

55:11

don't know emotionally like a good thing but I

55:15

guess that brings me to another question of you

55:17

know if you go to somewhere like diskprices.com

55:20

to see what the best deal might be they just

55:22

by default show you a bunch of used drives

55:25

and those seem to have great prices

55:27

and they seem like enterprise drives and I'm just

55:29

curious your opinion of you

55:32

know if you shouldn't care about drives they

55:34

should just be replaceable then is that

55:37

a reasonable avenue to go down?

55:39

If you're running a data center yes. If

55:42

you are a man with less

55:46

than a dozen hard drives in his house

55:49

then each one actually matters at least

55:52

a little bit. You could argue that an off-site backup

55:55

makes that argument a little bit less relevant

55:58

but for every time a drive fails, it's

56:00

extra cost, it's extra cognitive load, it's

56:03

extra time watching and re-silvering, and

56:06

all that kind of stuff. So from my

56:08

perspective, you're

56:10

going to be saving probably 30% or 40% with

56:12

a used drive versus a brand

56:14

new one. But you just don't

56:17

know its history. There's

56:19

just no way to know how many times Linus Tech Tips

56:21

has dropped that particular drive, is there?

56:24

Oh, local drives. Yeah,

56:27

I guess you're right. The same applies. If

56:29

I'm not comfortable with the history of the drives that I have in

56:31

my own care, then I certainly shouldn't

56:33

be comfortable with the drives

56:35

that I know not the history.

56:37

There are certain things in a computer that I will absolutely

56:40

advocate buying used.

56:42

RAM,

56:42

definitely, because you can check that really easily. Really?

56:45

Yeah. CPUs, they almost

56:47

have only ever had one go bad

56:49

on me. Motherboards, again,

56:52

they either typically work or they don't, as long as you check

56:54

the CPU pins very carefully. Power

56:56

supplies, typically, they're

56:59

pretty reliable, too, as long as they've got all the original

57:01

modular cables with them. You don't swap them

57:03

out with a different manufacturer, because the pinouts can be

57:05

wrong, and then wrong voltages fry

57:08

things. Obviously, a case, as

57:10

well. Obviously, that doesn't really matter if

57:12

that's used or not. But hard drives

57:14

are the one thing, because they are so

57:17

delicate, and the tolerances are so small, and

57:19

they're so fragile.

57:21

I just don't mess about with them. Well,

57:24

that's great advice. Thank you, Alex. It feels like

57:26

now I just got to open my wallet, see what I get

57:29

shipped here this week.

57:32

45homelab.com, the HL15

57:35

is here. It's available for purchase. You

57:37

can get it in a bare bones thing, or you can get

57:39

it built up with a chassis and a backplane

57:41

and the PSU, ready to go, fully tested.

57:44

It's really up to you. Big, strong, fast storage

57:46

for the Homelab. Taking the enterprise

57:49

mindset, the lessons learned that 45

57:51

Drives has taken from all their years in the enterprise

57:53

and collaborating with the open source community, and

57:56

now building something for the Homelabber.

57:58

It comes with Rocky Linux. you'll have the ability to

58:00

install applications on it. Things like Image will

58:03

run fantastic on the HL-15.

58:06

And from what I hear, Alex has one arriving

58:08

soon, or maybe already arrived, actually. I

58:11

don't want to say too much, but chances are there's

58:13

going to be a review on the show. I'm really excited

58:15

because now we have something that I think

58:17

meets the bar, and it takes this

58:19

as seriously as we all do. So go try

58:22

it out. I think this is the perfect unit for anybody that

58:24

wants something robust that can grow with them.

58:26

It's the HL-15, and you can find it over at 45homelab.com.

58:31

It's here. It's based on your feedback, and

58:33

it's beautiful, big, strong, and fast at 45homelab.com. Well,

58:38

Alex, thanks so much for your insights. As always,

58:41

I

58:42

should just listen to what you say and just go and do

58:44

it. So thank you for that. Oh,

58:46

stop. But I bet you we've got a whole bunch of

58:48

booths we should go through.

58:50

We have a nice batch this week. We can't always fit them

58:52

all for runtime.

58:54

But Bamham 182 comes

58:56

in with 34,567 SATs using

59:01

Podverse this week. And they write, I

59:03

appreciated hearing about the backups as I'm building

59:05

a backup server. I've gotten an Elite

59:07

Desk 800 G3 SFF for $40 on eBay. That's

59:13

amazing. That's what I keep saying

59:15

about these one-liter PCs, man. They

59:17

are bargain of the century. And

59:20

it has two 3.5 MVME drives in it. I

59:23

bought the Easy Stores, which will

59:25

replace some drives in my main zPool. These

59:28

will go into the Elite Desk. If all goes as

59:30

planned, the box will go to a friend's house and

59:32

is off all the time. But it'll turn on at night

59:34

and sync the data through WireGuard and then

59:37

turn back off. I switched from PFSense

59:39

to OpenSense with all of the WireGuard

59:41

drama, and then from OpenSense to

59:43

NixOS because I became more and more

59:45

familiar with NixOS. Absolutely

59:48

no regrets so far. I love having full control

59:50

and not having things I don't need running 24-7. I

59:53

recently wanted to swap out a Dell Wyze I

59:55

was using for a T740. I

59:58

was able to get the T740 running. with a

1:00:00

very similar config, swap the boxes in it

1:00:02

without the family even realizing I had

1:00:04

done it. Nice job, B.

1:00:07

Nice. I have a question here.

1:00:10

Given these little one liter PCs and they just

1:00:12

sip power, why would you bother

1:00:14

turning them on and off at night like that? It's

1:00:16

probably not your power bill. It's not your power, right?

1:00:18

So even a little bit, still something, right? Yeah,

1:00:21

but I just worry about

1:00:24

mechanical hard drives, spinning up, spinning down.

1:00:26

Is that a consideration here? Or am I just

1:00:29

worrying for no reason? It is a consideration.

1:00:31

It adds significant complexity

1:00:33

to all of your monitoring as well. Also

1:00:36

the exit conditions of the backup scripts

1:00:39

itself, it becomes a consideration

1:00:41

to you. If the server's on all

1:00:43

the time, you don't have to have any of those considerations.

1:00:46

You just can assume it's always on.

1:00:49

And if ZFS tries to do a replication, it will

1:00:52

say, well, you're already doing a replication dummy. Don't

1:00:55

try and do another one. It's in the middle of one and it shuts

1:00:57

the computer down. And then

1:01:00

let's say you just took a trip, hypothetical

1:01:02

example, and you're dumping 100 gigs worth

1:01:04

of pictures onto your local

1:01:07

system. That could easily take

1:01:09

a couple of days or more to upload to the remote

1:01:11

site. And in that case,

1:01:14

you've got to plan for that one-off

1:01:16

eventuality, that unusual eventuality. But

1:01:18

you've still got to plan for it. And like I say,

1:01:21

it just adds complexity to the whole situation.

1:01:24

And reading the rest of this comment, you're

1:01:26

clearly a gentleman that appreciates the finer

1:01:29

things in life by ditching any

1:01:32

non-declarity of OS. So

1:01:35

look at us go. I

1:01:39

love it. That's right, the gentleman's OS. I

1:01:42

think it's a pretty solid setup. I totally get

1:01:45

turning it on and off to save power,

1:01:47

although I have a similar setup here. I have

1:01:49

my main workstation

1:01:51

on a

1:01:52

Zigbee Smart Plug. When

1:01:55

I arrive, I have an automation that turns

1:01:58

on the Smart Plug and my BIOS is set. to

1:02:00

restore power when the power

1:02:02

state comes back. And that's worked

1:02:04

for six months. And then when I went on my trip

1:02:06

and came back, the smart plug turned on. And

1:02:09

now every other time, the PC doesn't restore

1:02:11

power state. Just for some reason, still

1:02:13

set in the BIOS, I haven't changed a thing,

1:02:16

and it just doesn't restore power state. So that kind of thing

1:02:18

can happen too.

1:02:19

Now, Alex, this gets me thinking about, you

1:02:22

sent me a photo a couple days ago of

1:02:25

something you 3D printed for your one liter PC,

1:02:27

which to me on the surface seems ridiculous.

1:02:30

Oh, yeah. But

1:02:32

can you maybe give us more details?

1:02:35

I 3D printed a rack mount, a

1:02:37

one new rack mount for these little systems. Chris,

1:02:39

I'm going to just see your listeners. Here we go. Oh,

1:02:43

my goodness. This is amazing. So I've

1:02:45

got one of these little Dell boxes, one of these little

1:02:47

PCs in a rack mount

1:02:49

system now.

1:02:51

That is so cool.

1:02:54

And then obviously, a 3D printer can't fit 19

1:02:56

inches, which is a standard rack unit width. So

1:02:59

you split it in half, put some screws in the middle,

1:03:01

and suddenly you've got a 19 inch wide

1:03:04

one unit for two of these things. You

1:03:06

put two of them in there, one, yeah, and one. Oh, my gosh.

1:03:09

There's quite a few if you look on like principles or Thingiverse

1:03:11

or something like that. There's quite a few people that sort

1:03:13

of stack them on top of each other like this, and you can

1:03:16

fit an eight port switch in there

1:03:18

as well. So you can actually literally have like a five or

1:03:20

six half width U

1:03:23

rack, like a mini rack almost,

1:03:25

like a desktop home lab rack, just

1:03:27

3D printing. I need that for the ODROID.

1:03:30

I have yet to find a case I am fully satisfied

1:03:33

with, with the ODROID. And I just need a rack

1:03:35

unit that the top is exposed. That's

1:03:38

really great. Just buy a bamboo lab printer, dude, and be done with

1:03:40

it and get it over with. All

1:03:42

right. Johnny Castaway comes in with 26,235 stats using podverse. Johnny

1:03:47

Castaway? Isn't that great? And that's a great name.

1:03:50

He's a member. That brings me back. That

1:03:53

brings me back. Longtime listener. And

1:03:56

he says thank you for contributing to everybody who makes

1:03:58

my eight hour weekly commute. We

1:04:01

are the we are the commuters friend. He

1:04:03

says I was once at peace with arch then I

1:04:05

tried Nick's OS gave up

1:04:07

many times returned back to my old arch slippers,

1:04:10

but Nick's kept taunting

1:04:12

me like a beast Nick's has been tamed

1:04:14

now, and I think it's here to stay He says

1:04:16

my next endeavor is Nick's OS on the steam

1:04:18

deck any thoughts No,

1:04:21

do you want to mess up a good thing? Oh, it's a

1:04:23

UK. He also comes in from the UK Yeah,

1:04:26

all of this learning people must do these

1:04:28

days, right? Yeah,

1:04:31

it's funny because I was such a curmudgeon

1:04:33

when it came to Nick's as well and Chris brought

1:04:35

it up in the show and I

1:04:37

believe privately said to me Alex if you just listen

1:04:40

sometimes I identified Nick's years

1:04:42

ago Why couldn't we've done this in self? I

1:04:44

was like yeah, but I just didn't see it and

1:04:46

it's just one of those things like nobody

1:04:48

nobody can tell you how

1:04:51

good Nick's is nobody can Can

1:04:53

tell you how it's gonna change your

1:04:55

life You just have to experience

1:04:58

in it just has to click For you

1:05:00

and it's like that moment as a teenager

1:05:02

where your parents tell you you just don't understand

1:05:05

You're like well one day eventually

1:05:07

with enough experience you

1:05:09

realize just how powerful

1:05:12

that stuff can be You

1:05:14

know in Linux unplugged sometimes we get a hard time because

1:05:16

we're mentioning Nick's OS Far

1:05:18

more this year than well we ever

1:05:21

had But

1:05:22

it's just

1:05:23

because it solves so many issues

1:05:26

in a beautiful way I mean, there's that learning

1:05:28

curve, of course, but I

1:05:30

have heard just this week alone

1:05:32

like five to six different

1:05:35

people say Oh, yeah.

1:05:37

Thanks guys. You made me try Nick's OS now. I can't

1:05:39

use anything else and So

1:05:41

if you haven't tried it yet, I I'm

1:05:44

running it on this laptop in front of me and I was a curmudgeon

1:05:46

too It took me a while to come to it and it just like it's

1:05:48

just kind of nice It's kind of nice

1:05:51

another tool in the toolbox. I mean you look at

1:05:53

the the wider ecosystem

1:05:55

I mean the whole flakes adoption situation

1:05:58

is frankly. It's just a mess But

1:06:00

you've got companies like Determinate Systems with

1:06:03

Flake Hub coming along, pushing the agenda, and

1:06:05

taking the project by the scruff of the neck and saying,

1:06:08

Come on boys, sort this out. That's

1:06:10

exactly what it needs. And for me,

1:06:12

those signs are the green

1:06:14

shoots that I needed to know that the project was

1:06:16

going to be worth investing in in the long term.

1:06:20

Yep, I completely agree. Although I'll take the other side

1:06:22

just to be fun and say, Don't try it because it'll wreck you. Stick

1:06:24

with your distro. Stick. Don't change it. You'll

1:06:26

never be the same. The Bitcoin

1:06:29

Dad Pod comes in with 24,444 sats using Podverse. It

1:06:33

writes, Backblaze Cloud

1:06:35

is another remote endpoint for backups. I use Restick

1:06:38

and their S3 encrypted object storage. My

1:06:40

last bill was $4 for just two

1:06:42

terabytes of storage. Whoa.

1:06:45

That's attractive. Mm-hmm.

1:06:46

Noted. It says I've deployed Tailscale

1:06:49

at limited scale and I find it amazing. That

1:06:51

said, I'm concerned about ReLion on it

1:06:53

for as a third party now that it has

1:06:55

visibility into my network. They also seem like

1:06:57

an obvious point of regulation in a world of governments

1:07:00

obsessed with surveilling private online activity. Has

1:07:02

anyone ever tried to migrate a Tailscale set up to Headscale?

1:07:06

What are your thoughts on the potential risks? Well,

1:07:08

I can't speak to the migration to Headscale because

1:07:10

I completely trust the Tailscale

1:07:13

infrastructure. And here's why. If

1:07:15

you know anything about public key

1:07:17

infrastructure, you will know that that is

1:07:19

what WireGuard is based on top of. So the

1:07:22

idea that you provide a key, the remote server

1:07:25

authenticates based on that kind of stuff, and

1:07:27

it's very difficult to break that

1:07:29

level of encryption. Any

1:07:32

data that transits any Tailscale

1:07:34

asset is encrypted using

1:07:37

that methodology. But

1:07:39

the thing about Tailscale that kind of makes it magic,

1:07:42

and okay, yes, I work for them. Corporate

1:07:44

shill moment, hat on, is

1:07:47

most of the connections happen through the magic

1:07:49

sock. So they become peer to peer connections.

1:07:52

So very little data actually traverses

1:07:55

any Tailscale owned infrastructure. Any

1:07:58

point you can look at Tailscale, scale status,

1:08:01

look at all the nodes in your network, you can do tail

1:08:04

scale net check, which prints an analysis

1:08:07

of the local network conditions,

1:08:09

like all the local DERP servers, all that kind of stuff.

1:08:12

So when you are connecting two tail

1:08:14

scale nodes together, they initially

1:08:17

connect and report back through

1:08:19

the tail scale control server via

1:08:21

this DERP mechanism, D-E-R-P.

1:08:24

The only thing that's identifiable on there is, you know, like an

1:08:26

IP address, that kind of stuff, but frankly,

1:08:29

that's identifiable across many other

1:08:31

situations than just tail scale. The

1:08:33

actual packets, the actual traffic? Tail

1:08:36

scale aren't interested in that, because that would cost them money

1:08:38

to transit through their servers, all that kind of stuff. And

1:08:41

if you look at the free tier that tail scale offers too,

1:08:43

those particular users that are on the

1:08:45

free tier, you think to yourself, well, how

1:08:48

can that possibly make sense from a business perspective

1:08:51

for them to offer this thing for free? Well,

1:08:53

it's because no data transits tail scales network,

1:08:55

and they're still using something like tail scale funnel or something

1:08:57

like that, and it's relayed through a TCP proxy.

1:09:00

But that's from like hosting a very simple website as

1:09:03

a developer or something like that. So,

1:09:05

you know, in terms of trusting them, there isn't much

1:09:07

trust, they actually need to give tail scale.

1:09:09

The only thing I need to give them is that they're going to continue

1:09:12

to exchange my wire guard keys for me. And

1:09:14

that's it really, that's the only trust that I need to put

1:09:17

in tail scale as a company and

1:09:19

to continue to exist, of course. Yeah, and

1:09:21

I also probably could have done more to play

1:09:23

around with head scale. But part of

1:09:26

the overall solution that I find super attractive

1:09:28

is that the hosted component where they

1:09:30

do that sort of derp discovery, I guess, would

1:09:33

be the term, which I love, is

1:09:36

sort of the value added that I needed to actually

1:09:38

start using it. I kept having

1:09:40

one off wire guard setups that would

1:09:43

I would have for six months or nine months at

1:09:45

a time, maybe a year at most, then

1:09:47

I'd rebuild it and set up in a different way. And I have a different

1:09:49

way to, you know, maybe even try to manage

1:09:51

it. And

1:09:53

that just,

1:09:54

that worked. But it was, it's

1:09:56

like it's a, it's a, it's a, it's a tail scale is

1:09:58

a different level of a solution. Because it brings

1:10:01

all my devices into one mesh network

1:10:04

And I was never gonna get to that level with

1:10:07

anything that I had to kind of self set

1:10:09

up like that It's just it just what I was never gonna spend

1:10:11

the time And so that's kind of that

1:10:13

extra value they brought there is what made me put

1:10:15

it on everything You could argue that switching

1:10:17

solutions every six months is pretty secure

1:10:20

because by the time any script kiddies are cool Talk with

1:10:22

you you're on to the next thing. Yeah, that's true.

1:10:24

That's true Now we do have more boosted

1:10:26

in making to the show because we had ten total boosters

1:10:29

this week But we'll put a link to the boost barn. We

1:10:31

do read all of them, and then we save them for posterity

1:10:34

So thank you everybody who boosts in we stacked

1:10:36

three hundred and forty six thousand eight hundred and

1:10:38

four sats this week we really appreciate

1:10:40

that you can boost in by getting a podcast

1:10:43

app at podcast apps calm like pod verse

1:10:45

and Cast-a-matic and

1:10:47

fountain those are really popular in our audience and

1:10:50

if you don't want to switch apps We'll just get out beat get albie

1:10:52

calm you top it off and head over to the podcast

1:10:54

index and boost and we'll have links To that and

1:10:57

also thank you to our members once again. Don't forget. We got that

1:10:59

black Friday So two bucks off

1:11:01

a month for a year. We use promo code black

1:11:03

Friday You can

1:11:06

find me at Alex dot Katie said me

1:11:08

got a bunch of links to the various places But

1:11:11

I'm on the internet over there one

1:11:13

place you can find me as Linux unplugged Linux

1:11:15

unplugged calm

1:11:17

Oh bunches shenanigans happening over there,

1:11:19

too

1:11:20

Yeah, you can find me over there sometimes not

1:11:22

every week though sometimes. It's evil Chris You

1:11:24

just have to figure out if it's good Chris or evil Chris, but this

1:11:26

is good Chris for sure Although if it was

1:11:29

evil Chris I wouldn't tell you but they're both tweeting

1:11:31

at Chris LAS on weapon X and the show

1:11:33

is also at self hosted show if you Want to follow that

1:11:35

over there? Thanks for listening everybody. That

1:11:37

was self hosted not show

Unlock more with Podchaser Pro

  • Audience Insights
  • Contact Information
  • Demographics
  • Charts
  • Sponsor History
  • and More!
Pro Features