0:02

Two and a half admins, episode 193. I'm Joe. I'm

0:05

Jim. And I'm Alan. And here we are

0:07

again. Is your PC having

0:09

trouble? Your smart TV might be to blame.

0:11

So this is a pretty interesting one. Somebody

0:14

discovered that they were having a whole bunch of

0:17

weird problems on their Windows PC.

0:20

Some settings weren't displaying, task managers sometimes wouldn't open.

0:22

It was very difficult to try to pin it

0:24

down to a root cause. And the

0:26

thing that it eventually got pinned down to was

0:29

oddly enough, their smart television.

0:32

They had a Hisense TV and the

0:34

issue is that the Hisense TV kept

0:37

spamming a different randomized UUID

0:39

every time it would pull

0:41

the network, essentially. And the

0:43

Windows machine was cashing all

0:45

of these UUIDs and eventually the database

0:47

of UUIDs that it encountered on the

0:50

network got so large that it caused

0:52

performance issues. This reminds me a

0:54

lot of the issues that I saw when I was

0:56

testing a home network monitoring product for Ars Technica. We're

0:58

an Android family in my house, obviously. And

1:02

when I first set up the monitoring system,

1:04

one of the things that it does is

1:06

it, again, it builds up a database of

1:08

all the, in this case, MAC addresses of

1:10

network devices that it can see. And

1:12

that allows you to figure out what those

1:14

devices are. You could assign labels to them.

1:17

So for devices that didn't broadcast

1:19

a host name, but just their MAC address,

1:21

you could identify it and then name it

1:23

like Janice's phone or what have you. The

1:26

problem was that eventually Janice brought some

1:28

friends over and those friends weren't Android

1:30

folks, they were iPhone folks. And

1:32

that was how I discovered that iPhones had begun

1:35

randomizing MAC addresses on

1:38

every network but their own home

1:40

network. And the impact there was that

1:42

all of a sudden my monitoring system

1:44

was giving me literally hundreds

1:46

of new device detected alerts

1:49

every single day because in

1:52

the course of somebody being over for an hour

1:54

or two to hang out, their iPhone would show

1:56

up as like 20 different devices because

1:59

of the randomized MAC addresses. addresses. So

2:01

seeing this issue with the database of

2:03

UUIDs on the Windows machine, it felt

2:06

very unpleasantly familiar. Also

2:08

probably worth noting, I have a high-sense television

2:11

myself and I would be worried about that

2:14

if I let the freaking thing on my network, which

2:16

I do not. Yeah, I also have

2:18

a high-sense TV and I have never once connected

2:20

to the network and I have no plans to.

2:22

Yeah, when we were talking about the Roku story

2:24

a couple weeks ago, I was like, yeah, I

2:26

have a Roku TV but it's not connected to

2:29

the internet. It's not allowed to talk to anybody

2:31

to have a new toss for me to have

2:33

to accept. It's the nice thing

2:35

about having a standalone Roku. You can

2:37

still be a pretty mainstream, normy type

2:39

and live an easy life and have

2:41

the purpose-designed device like a Roku or

2:43

other set-top box that gives you easy

2:46

access to the mainstream streaming channels. And

2:48

all that's fine. But if

2:50

you get that as a separate device from

2:52

the actual television, now if it

2:54

does something that you don't like, you have to replace

2:56

like a $60 box

2:58

rather than your potentially $1,200 television

3:01

set. Yeah,

3:03

exactly. Like I've got an Amazon

3:05

Firestick HD 4K, whatever it

3:07

is, Max. And yeah, if

3:09

I don't like that, I can just chuck it away

3:11

and I've got my Linux desktop connected to it as

3:14

well. Yeah, it's mine. I have a

3:16

little Beelink mini PC that runs the

3:18

screen. But yeah, like what I did

3:20

for my mom was took her old TV and got

3:22

her a Roku stick to plug into it because that

3:24

let her have what she wanted, which was

3:26

being able to stream stuff from my plaques in

3:28

her bedroom. I have both. My

3:30

current Kodi box is a repurposed

3:33

industrial controller that got retired. And

3:35

now it's living as my my

3:37

Kodi machine. That and my

3:39

Roku are both connected. And I just, you know, use the

3:42

TV remote to pick which one I want. The

3:44

other related one to this story is I

3:46

found something similar with Logitech's

3:48

G-Hub software on the gaming rigs

3:50

in my house. It

3:53

was trying to do something over the network and creating

3:55

rather a lot of traffic and a bunch

3:57

of CPUs in this like secondary

4:00

service it was running and I just

4:02

nuked it and deleted the executable and

4:05

the program still works fine and I

4:07

can control the LED lights in

4:09

my mouse but it's also not

4:11

gibbering on the network to every machine

4:13

in earshot about whatever it

4:15

was doing. I don't know why it

4:17

was using the network but it was creating a bunch of traffic for

4:20

no reason. But this high sense one

4:22

is interesting because the way it was effectively creating

4:24

the Java service was the PC

4:26

was tracking every one of those as a unique

4:28

TV that you could stream something to and

4:31

it just eventually a list of those got so long

4:33

that iterating over the list would use up a bunch

4:35

of CPU cycles to the point where when

4:37

you try to open display manager it's like oh do

4:39

you want to stream to one of this list of

4:41

a thousand TVs we've seen in your house and

4:45

rendering that list would make the menu take so long

4:47

open it would give up. Does this count

4:49

as an accidental denial of service? It is

4:51

basically high sense denial of servicing your PC

4:53

exactly. I might actually object a little bit

4:55

to the word accidental there because the thing

4:57

that's been kind of stewing in my head

4:59

as we've been talking about this is a

5:02

sort of a general complaint about the

5:04

industry in the last few years that

5:07

feels like devices have gotten a lot

5:09

more adversarial. Because when we talk about

5:11

whether it's a television randomizing UUIDs or

5:13

an iPhone randomizing MAC address in both

5:15

cases what that really is it's an

5:17

adversarial technique. It's saying I don't trust and I

5:19

don't like these networks I'm connecting to and I don't

5:21

want to give them information. And while

5:24

a lot of the time we

5:26

do actually have adversarial relationships with

5:28

networks that we need to join.

5:31

You know maybe you don't want to leak a bunch

5:33

of information to whatever random public Wi-Fi hotspot you happen

5:35

to be at. What I

5:38

don't feel like we have is a sense

5:40

that any of those devices are actually on

5:42

our side. Like it feels like the

5:44

devices are adversarial to each other in between vendors

5:46

and none of them really give that much of

5:48

a crap about us. If they

5:50

did you know they'd be asking us like you

5:52

know how do you want to treat this network.

5:54

Is this a friendly network or you know is

5:57

this a freaking Starbucks somewhere. And I think it's

5:59

a useful this. I wish more people

6:01

thought that way. This is exactly what Windows does

6:03

when you connect to a network, right? It asks,

6:05

is this a home or is this a public

6:07

network? Yeah, exactly. It asks, is it a public

6:09

or private network? And I don't know

6:11

if MAC address randomization is tied to that. I

6:13

think Windows can do that now. I don't know

6:16

if it ever gets automatically

6:18

enabled by itself. But that is exactly

6:20

the right distinction, which means that Joe

6:22

has put me in the uncomfortable place

6:25

of championing Windows as being, you know,

6:27

the one shining example of doing it

6:29

right. Well, let's fire

6:31

back at Windows and say, surely,

6:33

it should be able to handle

6:35

this situation better. Microsoft should have

6:38

taken account of this possibility. Well,

6:40

I think it's more the implementation

6:43

of DLNA on this TV is

6:45

wrong. It's supposed to have one unique

6:47

ID specific to that TV, not be, oh,

6:49

I'll just randomly generate a new one every

6:51

couple of minutes or whatever it was doing.

6:54

I guess they probably meant for it to generate

6:56

a random one once and then keep it, but

6:58

something went wrong and it just generated a random

7:00

one every time the service started. So every time

7:02

they power cycle their TV, it gets a new

7:04

ID and Windows remembers every TV it's ever seen,

7:07

which is probably fine when that number

7:09

is like 30 or something and

7:11

not fine when that number is thousands.

7:13

Yeah, that's what I'm saying. Surely Windows should

7:16

be smart enough to go, OK, well, let's

7:18

save 100 or... Yeah,

7:21

and when we see the 101st TV will delete

7:23

the TV we saw most recently. Yeah, exactly. Damn

7:27

it. You're making me do it again and

7:29

defend Microsoft. I think you're

7:31

expecting a little too much in the way

7:33

of prescience from them. That definitely seems like

7:35

something that should be thought of now that

7:37

we've seen a television do that, but I'm

7:40

not really going to blame them for not

7:42

thinking ahead of time that a television might

7:45

treat the network as adversarial and generate random

7:47

bogus UUIDs every time it connects. Yeah, well,

7:49

it kind of goes back to the old

7:52

saying that made the Internet was be liberal

7:54

in what you accept, but strict in what

7:56

you put out, right? As Far as like

7:58

RFCs Go. But it turns out

8:01

the right answer is to assume everybody on

8:03

the internet is a dick says going to

8:05

try to break their set. And

8:08

so maybe it is why were

8:11

many. It wasn't when they built

8:13

it, but nowadays more devices said

8:15

assume that the person who made

8:17

the other device was the most

8:19

underpaid, overworked developer possible who barely

8:22

knew what they were doing. I

8:24

get that that you you can't

8:26

anticipate every possible way somebody else

8:28

will break something, right? Plus, anything

8:30

that uses up a resource said have a

8:32

limit on even in that limit maybe should

8:35

be higher than you think they could ever

8:37

read, but probably does need to be an

8:39

upper bounds. Not just you can add as

8:41

many items says, listen to the computer breaks.

8:43

Now we finally get to the point where

8:45

I agree with you. However, I will again

8:47

fire back that there are so many things

8:50

in Windows they should be higher on the

8:52

priority list in this thread. I'm not upset

8:54

that they didn't proactively figure out that some

8:56

delusion somewhere would do this and fill up

8:58

a computers list of you ideas. I

9:01

don't want them to have thought harder about

9:03

that specific set by the should push our

9:05

sex with a some passers I surely. absolutely

9:07

you know maybe it little take his eyes

9:09

at somebody to make her a refill. I

9:12

did have that's a little aptitude run on

9:14

your computer, monitor your neighborhood, dreading open my

9:16

thighs and blasting all a computer so they

9:18

freeze by pretending to be a million different

9:20

Tv is that I'm I work ten years

9:22

ago but I don't think you find myself

9:24

from I find these days the i guess

9:26

most of the one most people's life items

9:28

Rise p Now so. Has at least some

9:31

kind of default here. And more importantly,

9:33

the routers aren't coming from the factory

9:35

with the same password on all of

9:37

them unless you change it. Almost every

9:39

manufacturer analysis is putting randomly generated in

9:41

a unique to the device password on

9:43

from the factory there. because it it

9:45

used to be, you could absolutely connectors

9:47

ninety nine percent of the netgear router

9:49

as you saw out there with the

9:51

yeah default wife I password will the

9:53

reset website like default password I com

9:55

or whatever you just type it in

9:58

a bottle numbers have motor. Third, The

10:00

Roeder and find out what the default was.

10:02

Haters out so could anybody else. Okay,

10:06

this episode is sponsored by Tales Go.

10:08

Go to tailscale.com/two Five

10:11

I. Tell. Scale is

10:13

an intuitive program of a way to manage

10:15

a private network. It. Zero Trust

10:17

Network access the every organization to

10:19

news and with Tales kills a

10:21

sale policies you can security control

10:24

access to devices and services with

10:26

Next and Network Access controls. Loads.

10:29

Of the light, not Linux on

10:31

the house. use tail scales all

10:33

it's including controlling thirty printers, promoting

10:35

into their relatives systems to support.

10:37

Controlling. Harm Assistant. And. Sending

10:39

that a snapshots to a site backup

10:41

locations. I got it off in

10:43

minutes and you can to. Support.

10:45

The South and check out tail scale for

10:48

yourself. Go to tailscale.com/two Five

10:50

A and try out Tell Scale

10:52

for free from two hundred devices

10:54

and three users with no credit

10:57

card required. That's. Tail

10:59

scale.com/two Five A.

11:03

Gym. You've got a wild theory about

11:05

ai and malware. I don't think it's

11:07

actually that wow To be honest with

11:09

you, that's the interesting thing about it.

11:11

So. Essentially, I think that we're gonna

11:13

be seeing a I'm Hour before much longer,

11:16

an actual model that looks to replicate itself

11:18

and does he know make new copies of

11:20

itself across the internet wherever it can find

11:22

insufficiently protected space and instantiate run new copies.

11:25

I think that this is going to happen

11:27

for the same reason that the More As

11:29

Warm happened way back in the day. I

11:32

mean, it was never a good idea for

11:34

somebody to sit down and write a virus.

11:36

The. First, viruses and ones that were created. We're

11:39

not created for financial gain. They were created

11:41

to see if he could do it. Doesn't

11:43

seem like a cool thing to do. And

11:45

sure enough, hey, now we have you know,

11:47

self replicating nowhere. I think that's going to

11:49

happen with a I pretty soon because we've

11:51

got to the point where the models I

11:53

think. Can. easily be

11:55

trained up to the level of like

11:58

a typical script kiddie So

12:00

now all we're really looking for is somebody

12:02

to have the bright idea and be willing

12:04

to do it like literally just to be

12:06

the first one that made the malware. Even

12:09

if nobody else knows about it to that one

12:11

person who does it that's gonna be

12:13

enough reason like this is cool what i

12:16

did. And once that happens once

12:18

that concept is out there in the world

12:20

and we've actually got a models that are.

12:23

Not only replicating themselves

12:26

but training themselves. It's

12:29

gonna be a similar situation as you

12:31

have you know in any proper biological ecosystem

12:33

you know you you've got things in

12:35

there that you don't like and they're

12:37

not easy to root out because they

12:39

change and they adapt and they protect themselves.

12:41

Is it good idea to even talk

12:43

about this publicly are you gonna give

12:45

people ideas or you just assuming that

12:47

people already have these ideas those ideas

12:49

are already out there for sure the world

12:52

did not need. Some brilliant person to

12:54

come up with the idea of the morris worm in

12:56

order for more is to actually write it. People

12:59

frequently have way too

13:01

high an estimation of

13:04

the value of unrealized ideas

13:07

unrealized ideas are cheap to pretty easy to

13:09

come up with. And

13:11

ultimately i don't think whoever

13:13

is gonna make this is going to need to

13:15

have heard me talking about it i think the

13:17

idea is really obvious. I

13:20

think the novel part right now is

13:22

that not many people and like mainstream

13:24

sense not many people have really thought

13:26

of it yet and that kind of

13:28

surprises me because it looks relatively obvious.

13:31

And it's worth pointing out that there

13:33

are an awful lot of like free

13:35

resources out there that you can stand

13:37

up small models on. You don't

13:39

have to spend any money or you know

13:41

break a credit card out of your pocket to

13:44

get some free space on google cloud to run

13:46

a freaking jupyter net book and stand

13:48

up a model that's doing you know text

13:50

generation image generation you know what have you.

13:53

It really doesn't seem at all

13:56

unlikely to me that like i said we're gonna get

13:59

some somebody's gonna train. model to a script

14:01

kitty level of capability and it's going

14:03

to do everything from find

14:05

free to set up accounts that

14:07

it can instantiate itself onto to

14:10

exploiting CVEs and getting

14:12

access to places where it shouldn't have

14:14

access to. The big

14:16

thing I think right now that limits

14:18

this idea is the sheer volume of

14:21

spaces that we don't actually have with

14:24

enough of the right kind of compute to

14:26

run a powerful model. But

14:28

that's becoming more and more common every day. I

14:30

mean everybody is pitching AI as the new big

14:32

thing that everybody ought to be doing and trying

14:34

to make it easier to get into it and

14:36

pushing it and promoting it. And

14:39

it's going to happen and I think

14:41

the follow-on impacts are going to be

14:43

pretty serious because imagine how much more

14:45

difficult it becomes to get into any

14:48

kind of internet computing from the

14:51

backend side when all of this

14:53

like, oh, it's free and it's easy to set up your thing

14:55

and get your feet wet is no longer possible. Because

14:58

you've got AI script kitties out there busily

15:01

trying to incest every last bit of

15:03

resource they can find that's insufficiently protected.

15:06

Yeah, and even the new Apple M4

15:08

chips are AI enabled and your

15:10

Windows laptops now come with a co-pilot

15:12

key on the keyboard. Right,

15:14

and especially when laptops are having built-in TPUs

15:17

or some kind of AI co-processor type

15:19

thing, then the malware is going to have

15:21

a place where on every infected machine,

15:23

it can get some of this resource instead

15:25

of having to try to trick Amazon

15:27

into giving it away for free or whatever.

15:30

Although again, I think the difficulty of tricking Amazon or

15:32

Google into giving it away for free is lower than

15:34

you're giving it credit for. Well, I

15:37

definitely know it's low. There's a new story we

15:39

didn't cover on the show about some guy getting

15:41

sued for having run up a couple million dollars

15:44

of bills at Amazon and Microsoft's

15:46

cloud with no money or

15:48

ever intended paying them. He

15:51

just convinced him to give him millions for free and he was mining

15:53

some stupid coin on it instead of even

15:55

doing something useful. He spent like

15:58

three million dollars in cloud bills. to

16:00

mine a million dollars in coin, but

16:03

he didn't pay the $3 million to the cloud people and

16:05

they're like, hey, where's our money? And

16:07

once we're looking at self-retrainable models doing this kind

16:09

of thing, again, it's not

16:12

hard to imagine not only

16:14

training a model to do this kind of thing, but training

16:16

a model to be able to look

16:18

at the list of CVEs and Google and

16:20

try to find exploit kits and see if

16:23

it can use them, that's,

16:25

I think, a lower bar than a lot

16:27

of things that people are actually using models

16:29

like chat GPT for in production right now.

16:32

Now, chat GPT, modern chat GPT is

16:34

enormous and that obviously limits its scope

16:36

and it's very easy to take the

16:38

contrarian viewpoint and say, why would

16:40

you do that with AI when traditional code can

16:42

do the same thing in so

16:44

much smaller of a footprint? And yeah, absolutely, but

16:46

it'll be the same reason we do everything

16:48

else with AI that we do with AI, because

16:51

it's really easy. And who cares

16:53

if it's wasteful because it was easy. Part

16:56

of the reason why phishing attacks

16:58

are normally not so easily detectable

17:00

is because they're low

17:02

effort and then partly because they don't want to spend

17:04

a lot of time on it and also because they're

17:07

trying to target the people that'll be fooled by it.

17:09

But if you can get AI to make up some

17:11

of these phishing scans for you, probably going to be

17:13

a lot more convincing than the ones

17:15

that the type of people who have no

17:18

graphical design skill to do it are doing.

17:20

If every fish is as high quality as a

17:22

spearfish, then things are going to get a lot

17:24

more comfortable. And when,

17:27

inevitably, the folks that are using these

17:29

things, it's not a case of the

17:31

AI is crafting it for you, but

17:33

the AI is just literally directly phishing

17:36

and spearphishing and handing you results later,

17:38

which, let's face it, that's what's

17:41

going to happen. You don't get into phishing

17:43

because you're into making a hard living the

17:45

honest way, right? That's just

17:47

not how that works. And

17:49

once you give these things autonomy, and once

17:52

you start saying, okay, well, it can do

17:54

unsupervised self-retraining to become more –

17:57

to remain effective

17:59

and applicable for longer, you're

18:01

going to lose control of it. And

18:03

when I say it's going to look

18:06

a lot like a biological ecosystem, I

18:08

mean yes, eventually we're going to have

18:10

digital organisms that fill the same role

18:12

in the same ways that parasites do

18:14

in our real ecosystem. Do we

18:16

want to have amoebas in our water supply?

18:18

No, we don't. But it turns out we

18:21

don't control them. They are a life form

18:23

and in the immortal words of Jeff

18:25

Goldblum, life finds a

18:27

way. Yes, like with fleas on cats and

18:29

dogs and the old flea treatments just don't

18:32

work anymore because the fleas have evolved. And

18:34

so there's only a few flea treatments left at work

18:36

and potentially I suppose we could

18:39

be in the same situation digitally. How desperately do

18:41

we want to get rid of the common cold?

18:43

Have we made any genuine

18:46

progress towards it? No, we have

18:48

not. Why haven't we? Because

18:50

it can change itself to adapt

18:52

and overcome whatever specific tailored

18:54

responses to what we come up with. Essentially

18:56

the cold that you got this year is

18:58

not only not the cold you got last

19:00

year, it's probably not the cold you got

19:03

a month or two before that. Now

19:05

granted also you really shouldn't be getting colds

19:07

that often and if you are, mask up.

19:09

I was going to say we did

19:11

manage to basically get rid of the common

19:13

cold for about a year there when everyone

19:15

masked up and stayed home. But that's

19:18

not really practical is it? And this makes me think what

19:21

the hell are we going to do about

19:23

this if this nightmarish reality comes

19:25

to pass? We're going to do the same thing that

19:27

we do right now in the biological ecosystem and that

19:29

answer is live with it. We're going to

19:31

do the best that we can to minimize the spread of

19:33

the things that we really don't like but

19:36

we're talking about an eventual world where

19:38

we do not have control of all

19:40

of our digital resources and effectively we

19:43

kind of can't. Because there's

19:45

just too much and there's too much that

19:47

depends on the system working the way that

19:49

it does at that moment to tear the

19:51

whole thing down to get a leg up

19:53

you know on the digital organisms

19:55

that you want to get rid of like are you

19:58

willing to take down the entirety of Amity? Amazon

20:00

to flush out a thing that's only

20:02

doing x percent of monetary damage. And

20:05

no, you're probably not. You're probably like, that would cost way

20:07

more money and I'm not going to do that, which

20:10

I'll remind you again is already

20:12

the thought process behind an awful

20:14

lot of the AI that we're

20:16

using productively right now. It's easy

20:18

and it's convenient. The answer

20:20

probably, sadly, is AI antibodies. We're

20:24

going to have AI fighting in the opposite direction, trying to find

20:26

and shut stuff down, looking at everybody's

20:28

usage pattern and be like, that one looks like another

20:30

AI trying to do something. And we

20:33

will lose control of some of those that will

20:35

turn into more malware AI. Yeah, or

20:37

some of it will just disable real

20:39

workloads, you know, bystanders shut by the

20:42

antivirus. It'll hallucinate.

20:44

It'll make bad decisions. It's going

20:46

to be just like introducing a

20:49

new life form into a local

20:51

ecosystem hoping to cut the

20:54

numbers in another one there. Well, there's never

20:56

any poor follow-on effects there, right? It's

20:58

literally the old lady who's followed a fly.

21:03

So what's the timeline on this, Jim? When is this

21:05

going to happen? Is it happening

21:08

already? I don't think it's happening

21:10

yet. My best guess

21:12

would be I think by

21:14

2030, we'll at least have seen a proof

21:16

of concept that escaped out into the wild,

21:19

something roughly similar scale to like

21:21

the Morris worm. I

21:23

think probably by about 2030, we're going to see

21:25

something along those lines. Now, again,

21:27

much like the Morris worm, it's not going

21:29

to change everything overnight. When the

21:31

Morris worm originally went out, it was like,

21:34

oh, well, that happened and that sucked, and

21:36

we fixed it, and that's not a problem

21:38

anymore. And it wasn't like everybody just immediately

21:40

went out and had antivirus because that happened

21:42

once, you know? Okay,

21:44

this episode is sponsored by Collide.

21:47

When you go through airport security, there's one line

21:49

where the TSA agent checks your ID and

21:51

another where a machine scans your back. The

21:54

same thing happens in enterprise security, But

21:56

instead of passengers and luggage, it's end users

21:59

and their devices. These. Days most

22:01

companies a pretty good to the first part of the

22:03

equation. Where they check user I didn't say. But.

22:06

Use a devices can roll right through

22:08

authentication with getting inspected at all. In.

22:10

Fact: A huge percentage of companies allow

22:13

unmanaged on trusted devices to access the

22:15

data. That means an employee can

22:17

login from a laptop that has it's firewall

22:19

turned off and hasn't been updated in six

22:21

months. Or. Worse, that laptop might

22:24

belong to a bad actor using

22:26

employee credentials. Collide. Finally, souls

22:28

to devise trust problem. Collide.

22:30

And shows that Know device can log into

22:32

your Octa protected apps unless it passes your

22:34

security checks. Plus. You can use

22:36

Collide on devices without Mdm like a

22:38

linux fleet contract to devices and every

22:40

B y O de phone and laptop.

22:42

Any company. So. Support the shower

22:45

and go to collide.com/to find a to

22:47

watch a demo and see how it

22:49

works. That. K O

22:51

L Id A.com/to size

22:53

A. This. Isn't free

22:56

consulting then the first as quick thank you

22:58

to everyone is for says with paypal Impatient

23:00

Really do appreciate that. If. You

23:02

want to join those people who can

23:04

go to To.five edmunds.com Support. And

23:06

remember that's various amounts on patrons and get

23:08

an advert free Rss feed as either Justice.

23:11

So all the shows in the like not

23:13

in a semi. And. If you want

23:15

to send any questions for gym alone are you

23:17

feedback is an email So a to.five up and

23:19

start com. Another. Poker been a

23:21

patron as he gets to. Which. Is what

23:24

Phoenix has done. They. right? I'm trying

23:26

to avoid using said assess native

23:28

encryption together with remote reputation to

23:30

an offsite backup and on trusted

23:32

location. The threats norio is

23:34

possible data tampering. And. Data test

23:36

at that on safe location. My.

23:39

Concept is to use locals that

23:41

a snapshot for rollback capabilities. And.

23:43

He's rustic, storing backups at the offsite

23:45

location on a simple set of has

23:48

dataset. The. Aim is to have

23:50

old and I said assess advanced is locally.

23:52

The. Avoid any issues with remote reputation

23:55

a native encryption while still having

23:57

an encrypted of sight back up

23:59

with snapshots. just not provided by

24:01

ZFS there. Any thoughts on this

24:03

setup? So essentially, Phoenix, it

24:05

sounds like you've got a perfectly reasonable

24:07

plan. Restick is a very well-known backup

24:09

and restore utility, well-respected. And

24:12

you're not doing anything odd

24:15

or unusual here. Essentially, ZFS kind of doesn't

24:17

matter at this point. You're just using Restick

24:19

to make your backups. And Restick offers the

24:21

feature set that you want, so

24:23

that all sounds fine to me as

24:25

long as your workload can actually handle

24:28

that. I wouldn't be able to

24:30

use Restick the way that you're talking about, because I

24:32

have too much data to backup too frequently, and it

24:34

just can't keep up any other way than block-level

24:37

replication, like what ZFS does. But if

24:39

you can get by with Restick, well,

24:41

you're good to go. The

24:43

one thing I will caution you about

24:46

is that in return for not exposing

24:48

yourself to potential issues with ZFS native

24:50

encryption, which has less battle testing,

24:53

I get the desire to avoid those potential

24:55

issues. But you're trading

24:58

that for the issues of significantly more

25:00

complex backup and restore operations. So

25:03

you need to practice your restores,

25:05

practice your restores, practice your

25:07

restores. Make sure

25:09

that your workload with Restick is capable

25:11

of matching your RPO and especially RTO

25:13

targets, and that you know how

25:15

to perform those restores, and that it all works

25:18

fine, and that's pretty much it. The

25:20

other thing is that since those restores are going to

25:22

be a lot more complex, you probably

25:24

need to practice them regularly,

25:27

where with ZFS, because it's

25:29

so much simpler, basically

25:31

once you get the hang of it, there's not

25:33

really a whole lot that you need to practice

25:35

and keep an edge on. Other

25:37

things I would say is no matter

25:39

what software you're using to backup ZFS,

25:42

make sure you're backing up from a

25:45

snapshot, not from the live file system. If

25:48

your backup is going to take more than one

25:50

second, then the files are going

25:52

to be changing while you're backing them up,

25:54

and you want to A, get a consistent

25:56

copy of each file, and ideally have

25:58

all the files be from the same file. point in

26:00

time. And so backing up from

26:02

the .ZFS.snapshot, .Navis.snapshot

26:04

location means that the backup you

26:07

just took is of all

26:09

the files as they were exactly at that

26:11

time. And then using different snapshots when you

26:13

do the next backup, the next backup, and so on. But

26:16

this way you make sure that what you're

26:18

backing up is actually a consistent view of

26:20

those files, not just all

26:22

the files while they were changing as ResTick

26:24

is trying to scan while the files keep

26:27

changing up from under it. And at this

26:29

point, allow me to jump in, dear listeners,

26:31

because I am positive. Some of you out

26:33

there are like, you know what? I've just

26:35

been using Rsync of my file system, no

26:37

snapshots, whatever, to backup for years, maybe decades.

26:39

It's always been fine. Like, what are you

26:41

on about? Well, it basically just

26:43

means you're not backing up something that

26:46

changes very frequently or is sensitive

26:48

to two files being of different

26:51

versions because the backup was taking

26:53

place while those files were being changed. As

26:56

an example, if you've ever administered a

26:58

database, for example, MySQL, you'll be aware

27:00

that there are index files as well

27:02

as the database files. Now, those

27:04

two need to be in sync. When one of them changes,

27:06

the other needs to change with it. And

27:09

if it takes you a couple hours a night to

27:11

run your backup, the odds are real good that at

27:13

some point, MySQL will have been changing

27:15

those files and you'll end up with

27:18

one version of the index file and

27:20

a different version of the IBDATA file.

27:23

And you won't be able to import that database or

27:25

it will be corrupted or broken or whatever. That's

27:27

the kind of issue that we're talking about. Yeah,

27:29

and it can be as simple as at the

27:32

power plant when I worked there, they

27:34

had these Excel documents that referred to

27:36

other Excel documents. So there's two

27:38

different XLS files that referred

27:40

to each other. And so if you backed up

27:42

one of them at three o'clock and one

27:44

of them at three thirty, then they wouldn't

27:46

work anymore because they referred to rows and

27:49

the other one that didn't exist yet or

27:51

no longer existed or whatever. And

27:53

it could just break everything. I mean

27:55

essentially we're just restating the issue as it is,

27:57

which is that all the files have not necessarily

27:59

been backed up. up at the same point in

28:01

time, but that can cause you other problems. For

28:04

example, it's not that uncommon to have somebody

28:06

be like, oh, well, I need to get

28:08

the file back in exactly the condition it

28:10

was in at this specific time

28:12

that I know of when the backup ran. And

28:15

I don't want it from a couple hours later. Well,

28:18

if you have a monolithic backup that

28:20

takes several hours to run, you

28:22

don't actually know what point in time

28:24

you're going to have that file in

28:26

that backup archive. In theory,

28:29

the timestamp on the file itself,

28:31

the metadata should be intact. So

28:33

you could restore that file and

28:35

check its timestamp. And maybe

28:38

you'd know, but then again,

28:40

maybe not, because all you really know for sure is the

28:42

last time it was modified. So you

28:45

know it wasn't modified after midnight, but do you

28:47

know if that is the version as it was

28:50

at midnight or one o'clock or two o'clock? No,

28:52

you really don't. And in some cases, that isn't

28:54

a problem. In other cases, it very much is.

28:57

And just in general, like we say, you always want

28:59

to have three copies of all your data, and probably

29:01

one of them should be in a different format anyway.

29:04

So even in this case, while you're

29:06

having Restic make it over to ZFS,

29:08

because you're not using ZFS replication to

29:10

see, it's a different ZFS, and so

29:12

not likely to hit the same problem

29:14

if there was a problem, not that

29:16

there is. Well, the other big thing

29:18

there is that you're not potentially exposing yourself

29:21

to a waffle-style replication-corruption bug,

29:23

where you have a corrupt file system

29:25

at the source, and by replicating to

29:27

the target, you also corrupt the target.

29:29

Like not just you wrote data that

29:31

was already corrupt, but like you corrupt

29:34

the whole target and make it

29:36

unmountable. ZFS has not

29:38

had like a really big disaster like

29:40

that, but other copy-on-write file systems, and

29:42

the most notorious one is Waffle, the

29:45

one behind NetApp filers, if I recall correctly. That

29:48

one had a really nasty bug, where you

29:50

could end up getting your SAN corrupted, and

29:53

if you let that thing replicate

29:55

to a backup device, another SAN,

29:58

it would corrupt that one the exact same way. and

30:00

now both your production and

30:02

your backup and all the archives,

30:04

everything is just trash because you

30:07

destroyed both entire file systems. But

30:10

in order for that to happen, the corrupted

30:12

file system really needs to have essentially

30:14

block level access to the other one.

30:16

Not just file level, block

30:18

level. And when you're replicating,

30:20

that's the case. When you're

30:22

just dumping essentially a big tarball over,

30:24

like Rustic, not so much. Right,

30:27

well we better get out of here then. Remember, show

30:29

at 2.5admins.com if you want to

30:31

send any questions or your feedback. You can

30:34

find me at jrust.com/mastodon. You can

30:36

find me at mercenariesysadmin.com. And I'm

30:38

Ad Alan Jude. We'll see you

30:40

next week.