biofs

I was part of a multidisciplinary team at GRAIL that developed machine learning models for cancer detection (among other projects). Many of us had prior expertise in bioinformatics, machine learning, or software engineering. Some were familiar with the R or Python ecosystems. A few had experience from big tech companies and a few were using git for the first time. We all used GRAIL’s internal tools, written in Go, for authentication, accessing clinical datasets, distributed computing, and much more.

With limited resources and lots to do, the team’s shared libraries were incomplete. Some datasets and file formats were only convenient to use through Go APIs, some only with Python or R, and several wrappers were duplicated. It was difficult to try new tools that might’ve been a good fit (like Pytorch or Julia).

biofs is a FUSE filesystem that translated proprietary datasets and file formats to common ones that any tool or programming language could read and write. It turned an M:N problem to 1:N (internal tool implementation : user’s preferred language/ecosystem). It’s named “bio” after one team’s code repository directory but the concept might be useful in other contexts, too.

UX

Users of biofs would find data, credentials, and more in a normal-looking files. They could browse in their shell, open them from any language or editor, etc. This took the place of language bindings (which had to be written, installed, and updated) or piping from command line tools (grail-file cat s3://grail-data/some/obj | ...).

/mnt/biofs
├── s3
│   ├── grail-data
│   │   └── data.zip
│   ├── grail-joshnewman
│   └── # Other accessible buckets
├── tickets  # Credentials fetched from ticket-server
│   └── eng
│       └── aws  # AWS credentials, auto-refreshed on open()
└── # Other data sources

biofs also allowed composable transformations, or alternate views, of underlying data. These were discovered and computed on-demand and composed arbitrarily. To express this flexibility to users, we invented a pattern: the views of the path /a/b/c are found under /a/b/.../c/ (see addfs for implementation).

/mnt/biofs/s3/grail-data/.../data.zip
└── unzip
    ├── README.txt
    ├── index.bin
    └── shard0

We thought ... is fairly unlikely to collide with real pathnames, the ellipsis intuitively suggests “alternate views”, and it’s default-hidden by ls, etc. This is consistent whether c is a file or directory, which is nice (we frequently used a columnar data format that was a directory on disk). But it feels a little unintuitive to jump up to c’s parent before adding ..., so it’s a compromise.

Easy extension

My colleague Yaz previously made a tool (gfs) that implemented FUSE operations for an existing abstraction over local and remote files. In biofs, I wanted more flexibility: we’d also want to access various internal data sources and run on-demand data transformations. To make extension easy, biofs is built around a simple filesystem abstraction called fsnode.T with variants for directories and files:

type T interface {
	Info() os.FileInfo
	FSNodeT()
}

type Parent interface {
	T
	Child(ctx context.Context, name string) (T, error)
	Children() Iterator
	// Also mutation operations, if supported.
}

type Leaf interface {
	T
	OpenFile(ctx context.Context, flag int) (fsctx.File, error)
}

New data sources define their directory tree and file contents according to these interfaces. Helper functions made it easy to implement these interfaces with slices or fixed collections, if the data are small. The S3 implementation is lazy and efficient for good performance.

fsnode.Leaf is like an “unopened” file. The standard io/fs didn’t have a direct representation for this; it specifies the data of a "io/fs".FS.Open(name) call but isn’t opened yet, and can be opened multiple times.

Transformations (of read-only trees) have signatures like func(T) T.

• unzipfs reads a ZIP file Leaf’s contents to expose them as an fsnode.Parent (on-demand: index first, then entries if needed).
• Another (non-public) one converted an internal columnar data file’s fsnode.Parent (directory containing files for each column/shard) into a standard Arrow file which was an fsnode.Leaf.

Data sources and transformations on fsnode.Ts compose flexibly. In principle, a user could access a ZIP archive on S3, and find a git repository in there, browse to some commit in the git history, and open a file in the repository at that revision.

fsnodefuse implements the actual FUSE filesystem operations for any input fsnode.T. It can “detect” capabilities from the underlying implementation: if an fsctx.File implements ioctx.ReadSeeker, we support FUSE seek operations, etc.

Inspiration

The largest inputs to many of our machine learning training runs were methylated DNA fragment data, which we stored in an optimized, sharded, columnar format (similar to PAM) called “fragment files”. Classifier training code written in Go could read these with a library call. In other analyses we’d often end up piping CSV from a command-line conversion tool but that’s clunky and often too slow.

I got frustrated with this situation (and am prone to software tooling rabbit holes) so I wrote a Python native module (with Cython) for reading fragment files. It defined an ad hoc memory layout and iterator interface with cgo and Cython glue. When I considered optimizing iteration performance (vectorization, reducing allocations, etc.), or adding bindings for Julia, I realized it might be better to translate the data to an Arrow schema instead of reinventing it poorly. Then, since Arrow has a file format too (much like its in-memory one), we could even skip the language bindings entirely and just use files.

FUSE offered a convenient layer to implement this translation and was familiar from earlier projects (Yaz made gfs for accessing S3 objects as files; I had made a toy FUSE filesystem for browsing historical git repository contents). And, we already did much of our analysis work on partly-managed Linux VMs (in EC2) so deploying a FUSE filesystem tool for the team seemed tractable.