Building a Distributed File System Like Google Drive
A distributed file system stores files across multiple machines while presenting a unified interface to users. Google Drive, Dropbox, and OneDrive all solve the same fundamental problems: reliable storage, synchronization across devices, conflict resolution, and global access. This article explores how these systems work.
Key sources: "Designing Data-Intensive Applications" by Martin Kleppmann, Google File System paper (Ghemawat, Gobioff, Leung), Dropbox engineering blog.
What Makes Distributed File Storage Hard
A local file system is straightforward: files are stored on a single disk. A distributed file system faces additional challenges:
- Durability: Disks fail. The system must replicate data across machines.
- Consistency: Two users editing the same file simultaneously must not lose data.
- Latency: Accessing a file from the other side of the world should be fast.
- Scalability: Storage should grow seamlessly as more files are added.
- Conflict resolution: When conflicts occur, the system must resolve them without data loss.
Architecture Overview
A distributed file system has three main layers:
text
Client App
│
▼
Metadata Service ─── Database (file tree, permissions)
│
▼
Storage Layer ──── Object Store (file content)
The metadata service tracks the file hierarchy: filenames, directories, sizes, permissions, and which storage nodes hold each file.
The storage layer stores the actual file content. Large files are split into chunks (typically 4-64 MB) and distributed across storage nodes.
Storage: Object Store
Most modern distributed file systems use an object store for file content. Object stores (like AWS S3, Google Cloud Storage) provide:
- Durability: Data is replicated across multiple availability zones
- Scalability: Petabytes of storage without manual sharding
- Simplicity: A key-value interface (GET, PUT, DELETE)
Files are split into fixed-size chunks. Each chunk is stored as a separate object with a unique key:
text
File: report.pdf (15 MB)
Chunk 1: report.pdf.chunk_1 (4 MB) → stored at objects/node1/report.pdf.1
Chunk 2: report.pdf.chunk_2 (4 MB) → stored at objects/node2/report.pdf.2
Chunk 3: report.pdf.chunk_3 (4 MB) → stored at objects/node3/report.pdf.3
Chunk 4: report.pdf.chunk_4 (3 MB) → stored at objects/node4/report.pdf.4
Chunking enables efficient storage and parallel upload/download. Large files are assembled from chunks on read.
Metadata Service
The metadata service maintains the file system tree:
/ (root)
├── /Documents/
│ ├── report.pdf (size, created, modified, permissions, chunk list)
│ └── notes.txt (size, created, modified, permissions, chunk list)
├── /Photos/
│ ├── vacation.jpg
│ └── sunset.png
└── /Shared/
└── presentation.pptx
This metadata is stored in a database. The database must be highly available and consistent. Google Drive uses a distributed database like Spanner. Dropbox uses a custom metadata store.
Metadata Operations
Path resolution: "/Documents/report.pdf" → look up each path component in the database. This is fast with proper indexing.
Permission checks: Does user Alice have read access to this file? Check ACL (access control list) or group membership.
Version tracking: Each file modification creates a new version. The metadata service stores version history so users can revert to previous versions.
Synchronization and Conflict Resolution
When a user modifies a file on their laptop while the same file is being modified on their phone, a conflict occurs. The system must detect and resolve this.
Detection via Version Vectors
Each file has a version vector — a set of (device, version) pairs. When a device modifies a file, it increments its own version counter:
text
Initial: {laptop: 0, phone: 0}
Laptop edits: {laptop: 1, phone: 0} — version 1
Phone edits: {laptop: 0, phone: 1} — version 1 (different device)
If the server receives both version 1 updates without seeing one subsume the other, it detects a conflict.
Resolution Strategies
| Strategy | Behavior | Example | |----------|----------|---------| | Last-writer-wins | The most recent modification wins | Simplest, may lose data | | Version merging | Both versions are saved and merged | Google Docs collaborative editing | | Keep both | Save the conflicted copy as a separate file | Dropbox creates "conflicted copy" files | | Three-way merge | Merge based on common ancestor | Git-style merge |
Most consumer cloud storage services use a combination:
- For Google Docs/Sheets: Operational transformation — concurrent edits are merged in real time
- For uploaded files: Last-writer-wins or keep both with "conflicted copy" naming
Delta Sync
Uploading a 1 GB file because you changed one paragraph is wasteful. Delta sync (also called binary diff) identifies what changed and only uploads the difference.
text
File V1: "The quick brown fox jumps over the lazy dog"
File V2: "The quick brown fox leaps over the lazy dog"
Delta: Replace "jumps" with "leaps" (5 bytes changes, not 44 bytes)
Dropbox uses a custom binary diff algorithm (similar to rsync) that breaks files into blocks and identifies which blocks changed. Only changed blocks are uploaded.
Client Architecture
The desktop/mobile client handles:
- Watching the local file system for changes (using OS file watchers like inotify or FSEvents)
- Uploading changes to the server
- Downloading changes from the server
- Resolving conflicts locally when possible
- Caching recently accessed files locally
The client also maintains a local database of file metadata so it can detect changes without querying the server for every file.
Scaling Considerations
| Component | Scaling Strategy | |-----------|-----------------| | Metadata service | Horizontal sharding by user ID or file path prefix | | Object storage | Built-in scaling from S3/GCS | | Upload/download | CDN for popular files, chunked parallel uploads | | Search | Dedicated search index (Elasticsearch) | | Notifications | Push notification service (FCM, APNs) |
Key Takeaways
- Distributed file systems separate metadata (file tree) from content (chunks in object storage).
- Large files are split into fixed-size chunks for efficient storage and parallel transfer.
- Version vectors detect concurrent modifications that create conflicts.
- Delta sync reduces bandwidth by uploading only changed parts of files.
- Conflict resolution is the hardest problem — strategies range from last-writer-wins to real-time merging.
- Google Drive, Dropbox, and OneDrive all share this fundamental architecture with different optimizations.
Design principle: Separate metadata from content. Metadata needs a consistent, transactional database. Content needs durable, scalable object storage.