Sample Quality Control

You can calculate quality control statistics on your variant data using Spark SQL functions, which can be expressed in Python, R, Scala, or SQL.

Each of these functions returns an array of structs containing metrics for one sample. If sample ids are including in the input DataFrame, they will be propagated to the output. The functions assume that the genotypes in each row of the input DataFrame contain the same samples in the same order.

Functions

Arguments

Return

sample_call_summary_stats

referenceAllele string, alternateAlleles array of strings, genotypes array calls

A struct containing the following summary stats:

  • callRate: The fraction of variants where this sample has a called genotype. Equivalent to nCalled / (nCalled + nUncalled)

  • nCalled: The number of variants where this sample has a called genotype

  • nUncalled: The number of variants where this sample does not have a called genotype

  • nHomRef: The number of variants where this sample is homozygous reference

  • nHet: The number of variants where this sample is heterozygous

  • nHomVar: The number of variants where this sample is homozygous non reference

  • nSnv: The number of calls where this sample has a single nucleotide variant. This value is the sum of nTransition and nTransversion

  • nInsertion: Insertion variant count

  • nDeletion: Deletion variant count

  • nTransition: Transition count

  • nTransversion: Transversion count

  • nSpanningDeletion: The number of calls where this sample has a spanning deletion

  • rTiTv: Ratio of transitions to tranversions (nTransition / nTransversion)

  • rInsertionDeletion: Ratio of insertions to deletions (nInsertion / nDeletion)

  • rHetHomVar: Ratio of heterozygous to homozygous variant calls (nHet / nHomVar)

sample_dp_summary_stats

genotypes array with a depth field

A struct with min, max, mean, and stddev

sample_gq_summary_stats

genotypes array with a conditionalQuality field

A struct with min, max, mean, and stddev

Computing user-defined sample QC metrics

In addition to the built-in QC functions discussed above, Glow provides two ways to compute user-defined per-sample statistics.

Explode and aggregate

If your dataset is not in a normalized, pVCF-esque shape, or if you want the aggregation output in a table rather than a single array, you can explode the genotypes array and use any of the aggregation functions built into Spark. For example, this code snippet computes the number of sites with a non-reference allele for each sample:

import pyspark.sql.functions as fx
exploded_df = df.withColumn("genotype", fx.explode("genotypes"))\
  .withColumn("hasNonRef", fx.expr("exists(genotype.calls, call -> call != -1 and call != 0)"))

agg = exploded_df.groupBy("genotype.sampleId", "hasNonRef")\
  .agg(fx.count(fx.lit(1)))\
  .orderBy("sampleId", "hasNonRef")

Notebook