Jack Vanlightly @vanlightly

Likes breaking ideas and systems, writing, picking systems apart @confluentinc Ex @Splunk, @VMware https://t.co/voJWmL5iBE Credit: ESO/B. Tafreshi jack-vanlightly.com Barcelona, Spain Joined November 2016

Tweets

2K
Followers

5K
Following

251
Likes

1K

Jack Vanlightly @vanlightly

a week ago

My TLA+ spec of Chroma's S3 impl of wal3 is done. Next weekend, the write-up where I'll find commonalities and differences across the surveyed WAL-on-S3 designs, maybe come up with some kind of taxonomy. I might come back later and cover the rest of the designs in my todo list, assuming the Interest Goblin is still interested by then. github.com/Vanlightly/s3-…

3 6 45 3K 18

View Details

Jack Vanlightly @vanlightly

a week ago

I think I've largely failed to make the wal3 spec generalized and abstract. I think I went too far trying to make it faithful to the implementation and so it has become a monster. Still, it did detect a real bug. I can't spend more time making it beautiful though. Day job. Will publish it over the weekend, then I'm calling it a day on writing more specs. I'll do the final write-up and classification of all the WAL-on-S3 designs I have surveyed.

0 0 19 1K 5

View Details

Jack Vanlightly @vanlightly

a week ago

Yeah, it's really simple! Some of these WALs though are designed the way they are (I assume) because they are tightly integrated into the system they serve, so might not be great general purpose approaches. Some complexity may arise due to the specific demands of a specific system.

0 0 1 37 0

View Details

Jack Vanlightly @vanlightly

a week ago

Well, I got there too late. I just found a correctness bug in Chroma's wal3 this morning, and was working on an integration test to prove what my TLA+ spec found, and on reading the fork, I see that just 2 days ago, the bug got fixed! github.com/chroma-core/ch…

1 0 11 781 0

View Details

Jack Vanlightly @vanlightly

a week ago

@kellabyte Love the posts 😁

0 0 1 85 0

View Details

Jack Vanlightly @vanlightly

2 weeks ago

Next up, Chroma's wal3. Should be an interesting write up when I stop collecting designs.

1 1 12 1K 2

View Details

Jack Vanlightly @vanlightly

2 weeks ago

Description of the consistency buy and solution github.com/Vanlightly/s3-…

0 1 3 286 1

View Details

Jack Vanlightly @vanlightly

2 weeks ago

Another day, another WAL-on-S3 system encoded in a TLA+ spec. This one is Shared Stream Consensus (S2C) which is more of a shared log for SMR, but fits closely enough to a WAL. After all, a WAL on shared storage is a shared log. My spec confirmed a theoretical consistency bug in the design, related to GC freeing a log address so a stale writer could do a put-if-absent of a log entry behind the snapshot boundary. A second spec adds a fencing record in the log, SlateDB style, to avoid the scenario. github.com/Vanlightly/s3-…

4 5 31 2K 14

View Details

Jack Vanlightly @vanlightly

2 weeks ago

From the OSDI LogDrive paper usenix.org/system/files/o…

0 1 32 2K 19

View Details

Jack Vanlightly @vanlightly

2 weeks ago

cc @Mo69796244, I'll let you know when you can review ths spec and the variant with the fix.

1 0 1 341 0

View Details

Jack Vanlightly @vanlightly

2 weeks ago

I'm documenting the fix for the consistency issue I found in Shared Storage Consensus (S2C), and I'm referencing SlateDB (which I copied the solution from). That's what is neat about learning how varied systems work, you can easily steal what you already know works somewhere else! Will publish the latest spec over the weekend after more model checking.

1 1 22 1K 5

View Details

Jack Vanlightly @vanlightly

2 weeks ago

@DominikTornow It's something old from aeronautics. Some gauges are in triplicate as with two copies of the gauge one faulty value means the pilot can't tell which is the right one, but with three, the pilot can hopefully tell the one that is faulty.

0 0 1 56 0

View Details

Jack Vanlightly @vanlightly

3 weeks ago

@StephanEwen Congratulations!

0 0 2 279 0

View Details

Jack Vanlightly @vanlightly

3 weeks ago

When reverse engineering a TLA+ spec from code, the hardest thing is understanding the code. While AI can write syntactically correct TLA+, I can't say it's good at writing specs. So I write the spec and use AI to help me review it. But AI is really useful when reading the code. It helps me understand what each class does, the roles of certain variables and structures, things that without AI can be quite time consuming and laborious. But I still read the code, I want a good handle for its structure and organization. Then bit by bit, as I walk the flows in the code, I build up a more simplified model in TLA+. My goal in the end is understanding.

5 1 25 1K 8

View Details

Jack Vanlightly @vanlightly

3 weeks ago

@Mo69796244 I need to write the spec so I can get a firmer handle on it. I can also explore mitigations in the spec.

0 0 1 95 0

View Details

Jack Vanlightly @vanlightly

3 weeks ago

This morning I decided to look at Shared Storage Consensus (S2C) and one glaring correctness bug already stands out before I have started writing a spec. GC can delete log segments that allow a stale writer to make progress (resulting in unreadable data aka data loss). SlateDB uses fencing records which are not GCed for this purpose. S2C does not have this and so GC introduces a consistency problem. Whenever the WAL is defined by a sequential address space guarded by put-if-absent, GC introduces the risk of allowing a stale writer to make progress resulting in data loss.

1 1 23 1K 6

View Details

Jack Vanlightly @vanlightly

3 weeks ago

Right, in the case of a timeout, it would be invalid but harmless. It's a theoretical case. It would need very aggressive GC to result in the loss of an acknowledged write. The solution is to ensure that GC isn't that aggressive. SlateDB has the same issue with fence records. They currently don't GC them at all or ensure that GC of fence records leaves them in place for a long time.

1 0 1 106 0

View Details

Jack Vanlightly @vanlightly

3 weeks ago

From what I understand. If replica r1 is the leader and pauses after advancing the commitIndex to k (and validating it is still the leader) but before writing its batch, then a leader change could result in another node, r2, writing its own batch to k. Then GC kicks in, after a snapshot, deletes k. Then r1 resumes, writes its batch to k and it is a successful but invalid write. It's a small window, unlikely to happen. I'll need to confirm with my spec that my understanding is correct.