Getting Started

Running Locally

Glow requires Apache Spark 2.4.3 or later.

If you don’t have a local Apache Spark installation, you can install it from PyPI:

pip install pyspark==3.0.0

or download a specific distribution.

Install the Python frontend from pip:

pip install

and then start the Spark shell with the Glow maven package:

./bin/pyspark --packages io.projectglow:glow-spark3_2.12:1.0.1 --conf

To start a Jupyter notebook instead of a shell:

PYSPARK_DRIVER_PYTHON=jupyter PYSPARK_DRIVER_PYTHON_OPTS=notebook ./bin/pyspark --packages io.projectglow:glow-spark3_2.12:1.0.1 --conf

And now your notebook is glowing! To access the Glow functions, you need to register them with the Spark session.

import glow
spark = glow.register(spark)
df ='vcf').load(path)

Notebooks embedded in the docs

To demonstrate example use cases of Glow functionalities, most doc pages are accompanied by embedded Databricks Notebooks. Most of the code in these notebooks can be run on Spark and Glow alone, but a few functions such as display() or dbutils() are only available on Databricks. See Databricks notebooks for more info.

Also note that the path to datasets used as example in these notebooks is usually a folder in /databricks-datasets/genomics/ and should be replaced with the appropriate path based on your own folder structure.

Demo notebook

This notebook showcases some of the key functionality of Glow, like reading in a genomic dataset, saving it as a Delta Lake, and performing a genome-wide association study.