sourmash: working with private collections of signatures

Running this notebook.

You can run this notebook interactively via mybinder; click on this button: Binder

A rendered version of this notebook is available at sourmash.readthedocs.io under “Tutorials and notebooks”.

You can also get this notebook from the doc/ subdirectory of the sourmash github repository. See binder/environment.yaml for installation dependencies.

What is this?

This is a Jupyter Notebook using Python 3. If you are running this via binder, you can use Shift-ENTER to run cells, and double click on code cells to edit them.

Contact: C. Titus Brown, ctbrown@ucdavis.edu. Please file issues on GitHub if you have any questions or comments!

download a bunch of genomes

[1]:
!mkdir -p big_genomes
!curl -L https://osf.io/8uxj9/?action=download | (cd big_genomes && tar xzf -)
/Users/t/dev/sourmash/doc/big_genomes
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   459  100   459    0     0   1017      0 --:--:-- --:--:-- --:--:--  1017
100 61.1M  100 61.1M    0     0  2932k      0  0:00:21  0:00:21 --:--:-- 3468k

compute signatures for each file

[2]:
!cd big_genomes/ && sourmash sketch dna -p k=31,scaled=1000 --name-from-first *.fa
/Users/t/dev/sourmash/doc/big_genomes

== This is sourmash version 4.0.0a4.dev12+g31c5eda2. ==
== Please cite Brown and Irber (2016), doi:10.21105/joss.00027. ==

computing signatures for files: 0.fa, 1.fa, 10.fa, 11.fa, 12.fa, 13.fa, 14.fa, 15.fa, 16.fa, 17.fa, 18.fa, 19.fa, 2.fa, 20.fa, 21.fa, 22.fa, 23.fa, 24.fa, 25.fa, 26.fa, 27.fa, 28.fa, 29.fa, 3.fa, 30.fa, 31.fa, 32.fa, 33.fa, 34.fa, 35.fa, 36.fa, 37.fa, 38.fa, 39.fa, 4.fa, 40.fa, 41.fa, 42.fa, 43.fa, 44.fa, 45.fa, 46.fa, 47.fa, 48.fa, 49.fa, 5.fa, 50.fa, 51.fa, 52.fa, 53.fa, 54.fa, 55.fa, 56.fa, 57.fa, 58.fa, 59.fa, 6.fa, 60.fa, 61.fa, 62.fa, 63.fa, 7.fa, 8.fa, 9.fa
Computing a total of 1 signature(s).
... reading sequences from 0.fa
calculated 1 signatures for 1 sequences in 0.fa
saved signature(s) to 0.fa.sig. Note: signature license is CC0.
... reading sequences from 1.fa
calculated 1 signatures for 1 sequences in 1.fa
saved signature(s) to 1.fa.sig. Note: signature license is CC0.
... reading sequences from 10.fa
calculated 1 signatures for 1 sequences in 10.fa
saved signature(s) to 10.fa.sig. Note: signature license is CC0.
... reading sequences from 11.fa
calculated 1 signatures for 1 sequences in 11.fa
saved signature(s) to 11.fa.sig. Note: signature license is CC0.
... reading sequences from 12.fa
calculated 1 signatures for 1 sequences in 12.fa
saved signature(s) to 12.fa.sig. Note: signature license is CC0.
... reading sequences from 13.fa
calculated 1 signatures for 1 sequences in 13.fa
saved signature(s) to 13.fa.sig. Note: signature license is CC0.
... reading sequences from 14.fa
calculated 1 signatures for 1 sequences in 14.fa
saved signature(s) to 14.fa.sig. Note: signature license is CC0.
... reading sequences from 15.fa
calculated 1 signatures for 1 sequences in 15.fa
saved signature(s) to 15.fa.sig. Note: signature license is CC0.
... reading sequences from 16.fa
calculated 1 signatures for 4 sequences in 16.fa
saved signature(s) to 16.fa.sig. Note: signature license is CC0.
... reading sequences from 17.fa
calculated 1 signatures for 2 sequences in 17.fa
saved signature(s) to 17.fa.sig. Note: signature license is CC0.
... reading sequences from 18.fa
calculated 1 signatures for 1 sequences in 18.fa
saved signature(s) to 18.fa.sig. Note: signature license is CC0.
... reading sequences from 19.fa
calculated 1 signatures for 9 sequences in 19.fa
saved signature(s) to 19.fa.sig. Note: signature license is CC0.
... reading sequences from 2.fa
calculated 1 signatures for 1 sequences in 2.fa
saved signature(s) to 2.fa.sig. Note: signature license is CC0.
... reading sequences from 20.fa
calculated 1 signatures for 1 sequences in 20.fa
saved signature(s) to 20.fa.sig. Note: signature license is CC0.
... reading sequences from 21.fa
calculated 1 signatures for 1 sequences in 21.fa
saved signature(s) to 21.fa.sig. Note: signature license is CC0.
... reading sequences from 22.fa
calculated 1 signatures for 1 sequences in 22.fa
saved signature(s) to 22.fa.sig. Note: signature license is CC0.
... reading sequences from 23.fa
calculated 1 signatures for 5 sequences in 23.fa
saved signature(s) to 23.fa.sig. Note: signature license is CC0.
... reading sequences from 24.fa
calculated 1 signatures for 3 sequences in 24.fa
saved signature(s) to 24.fa.sig. Note: signature license is CC0.
... reading sequences from 25.fa
calculated 1 signatures for 1 sequences in 25.fa
saved signature(s) to 25.fa.sig. Note: signature license is CC0.
... reading sequences from 26.fa
calculated 1 signatures for 1 sequences in 26.fa
saved signature(s) to 26.fa.sig. Note: signature license is CC0.
... reading sequences from 27.fa
calculated 1 signatures for 1 sequences in 27.fa
saved signature(s) to 27.fa.sig. Note: signature license is CC0.
... reading sequences from 28.fa
calculated 1 signatures for 3 sequences in 28.fa
saved signature(s) to 28.fa.sig. Note: signature license is CC0.
... reading sequences from 29.fa
calculated 1 signatures for 1 sequences in 29.fa
saved signature(s) to 29.fa.sig. Note: signature license is CC0.
... reading sequences from 3.fa
calculated 1 signatures for 1 sequences in 3.fa
saved signature(s) to 3.fa.sig. Note: signature license is CC0.
... reading sequences from 30.fa
calculated 1 signatures for 1 sequences in 30.fa
saved signature(s) to 30.fa.sig. Note: signature license is CC0.
... reading sequences from 31.fa
calculated 1 signatures for 1 sequences in 31.fa
saved signature(s) to 31.fa.sig. Note: signature license is CC0.
... reading sequences from 32.fa
calculated 1 signatures for 1 sequences in 32.fa
saved signature(s) to 32.fa.sig. Note: signature license is CC0.
... reading sequences from 33.fa
calculated 1 signatures for 1 sequences in 33.fa
saved signature(s) to 33.fa.sig. Note: signature license is CC0.
... reading sequences from 34.fa
calculated 1 signatures for 1 sequences in 34.fa
saved signature(s) to 34.fa.sig. Note: signature license is CC0.
... reading sequences from 35.fa
calculated 1 signatures for 7 sequences in 35.fa
saved signature(s) to 35.fa.sig. Note: signature license is CC0.
... reading sequences from 36.fa
calculated 1 signatures for 1 sequences in 36.fa
saved signature(s) to 36.fa.sig. Note: signature license is CC0.
... reading sequences from 37.fa
calculated 1 signatures for 1 sequences in 37.fa
saved signature(s) to 37.fa.sig. Note: signature license is CC0.
... reading sequences from 38.fa
calculated 1 signatures for 1 sequences in 38.fa
saved signature(s) to 38.fa.sig. Note: signature license is CC0.
... reading sequences from 39.fa
calculated 1 signatures for 1 sequences in 39.fa
saved signature(s) to 39.fa.sig. Note: signature license is CC0.
... reading sequences from 4.fa
calculated 1 signatures for 1 sequences in 4.fa
saved signature(s) to 4.fa.sig. Note: signature license is CC0.
... reading sequences from 40.fa
calculated 1 signatures for 1 sequences in 40.fa
saved signature(s) to 40.fa.sig. Note: signature license is CC0.
... reading sequences from 41.fa
calculated 1 signatures for 1 sequences in 41.fa
saved signature(s) to 41.fa.sig. Note: signature license is CC0.
... reading sequences from 42.fa
calculated 1 signatures for 1 sequences in 42.fa
saved signature(s) to 42.fa.sig. Note: signature license is CC0.
... reading sequences from 43.fa
calculated 1 signatures for 1 sequences in 43.fa
saved signature(s) to 43.fa.sig. Note: signature license is CC0.
... reading sequences from 44.fa
calculated 1 signatures for 2 sequences in 44.fa
saved signature(s) to 44.fa.sig. Note: signature license is CC0.
... reading sequences from 45.fa
calculated 1 signatures for 1 sequences in 45.fa
saved signature(s) to 45.fa.sig. Note: signature license is CC0.
... reading sequences from 46.fa
calculated 1 signatures for 1 sequences in 46.fa
saved signature(s) to 46.fa.sig. Note: signature license is CC0.
... reading sequences from 47.fa
calculated 1 signatures for 2 sequences in 47.fa
saved signature(s) to 47.fa.sig. Note: signature license is CC0.
... reading sequences from 48.fa
calculated 1 signatures for 1 sequences in 48.fa
saved signature(s) to 48.fa.sig. Note: signature license is CC0.
... reading sequences from 49.fa
calculated 1 signatures for 228 sequences in 49.fa
saved signature(s) to 49.fa.sig. Note: signature license is CC0.
... reading sequences from 5.fa
calculated 1 signatures for 1 sequences in 5.fa
saved signature(s) to 5.fa.sig. Note: signature license is CC0.
... reading sequences from 50.fa
calculated 1 signatures for 1 sequences in 50.fa
saved signature(s) to 50.fa.sig. Note: signature license is CC0.
... reading sequences from 51.fa
calculated 1 signatures for 1 sequences in 51.fa
saved signature(s) to 51.fa.sig. Note: signature license is CC0.
... reading sequences from 52.fa
calculated 1 signatures for 1 sequences in 52.fa
saved signature(s) to 52.fa.sig. Note: signature license is CC0.
... reading sequences from 53.fa
calculated 1 signatures for 1 sequences in 53.fa
saved signature(s) to 53.fa.sig. Note: signature license is CC0.
... reading sequences from 54.fa
calculated 1 signatures for 1 sequences in 54.fa
saved signature(s) to 54.fa.sig. Note: signature license is CC0.
... reading sequences from 55.fa
calculated 1 signatures for 1 sequences in 55.fa
saved signature(s) to 55.fa.sig. Note: signature license is CC0.
... reading sequences from 56.fa
calculated 1 signatures for 1 sequences in 56.fa
saved signature(s) to 56.fa.sig. Note: signature license is CC0.
... reading sequences from 57.fa
calculated 1 signatures for 1 sequences in 57.fa
saved signature(s) to 57.fa.sig. Note: signature license is CC0.
... reading sequences from 58.fa
calculated 1 signatures for 30 sequences in 58.fa
saved signature(s) to 58.fa.sig. Note: signature license is CC0.
... reading sequences from 59.fa
calculated 1 signatures for 5 sequences in 59.fa
saved signature(s) to 59.fa.sig. Note: signature license is CC0.
... reading sequences from 6.fa
calculated 1 signatures for 76 sequences in 6.fa
saved signature(s) to 6.fa.sig. Note: signature license is CC0.
... reading sequences from 60.fa
calculated 1 signatures for 11 sequences in 60.fa
saved signature(s) to 60.fa.sig. Note: signature license is CC0.
... reading sequences from 61.fa
calculated 1 signatures for 47 sequences in 61.fa
saved signature(s) to 61.fa.sig. Note: signature license is CC0.
... reading sequences from 62.fa
calculated 1 signatures for 1 sequences in 62.fa
saved signature(s) to 62.fa.sig. Note: signature license is CC0.
... reading sequences from 63.fa
calculated 1 signatures for 4 sequences in 63.fa
saved signature(s) to 63.fa.sig. Note: signature license is CC0.
... reading sequences from 7.fa
calculated 1 signatures for 3 sequences in 7.fa
saved signature(s) to 7.fa.sig. Note: signature license is CC0.
... reading sequences from 8.fa
calculated 1 signatures for 1 sequences in 8.fa
saved signature(s) to 8.fa.sig. Note: signature license is CC0.
... reading sequences from 9.fa
calculated 1 signatures for 3 sequences in 9.fa
saved signature(s) to 9.fa.sig. Note: signature license is CC0.

Compare them all

[3]:
!sourmash compare big_genomes/*.sig -o compare_all.mat
!sourmash plot compare_all.mat

== This is sourmash version 4.0.0a4.dev12+g31c5eda2. ==
== Please cite Brown and Irber (2016), doi:10.21105/joss.00027. ==

loaded 1 sigs from 'big_genomes/0.fa.sig'g'
loaded 1 sigs from 'big_genomes/1.fa.sig'g'
loaded 1 sigs from 'big_genomes/10.fa.sig'g'
loaded 1 sigs from 'big_genomes/11.fa.sig'g'
loaded 1 sigs from 'big_genomes/12.fa.sig'g'
loaded 1 sigs from 'big_genomes/13.fa.sig'g'
loaded 1 sigs from 'big_genomes/14.fa.sig'g'
loaded 1 sigs from 'big_genomes/15.fa.sig'g'
loaded 1 sigs from 'big_genomes/16.fa.sig'g'
loaded 1 sigs from 'big_genomes/17.fa.sig'10 sigs total
loaded 1 sigs from 'big_genomes/18.fa.sig'g'
loaded 1 sigs from 'big_genomes/19.fa.sig'g'
loaded 1 sigs from 'big_genomes/2.fa.sig'g'
loaded 1 sigs from 'big_genomes/20.fa.sig'g'
loaded 1 sigs from 'big_genomes/21.fa.sig'g'
loaded 1 sigs from 'big_genomes/22.fa.sig'g'
loaded 1 sigs from 'big_genomes/23.fa.sig'g'
loaded 1 sigs from 'big_genomes/24.fa.sig'g'
loaded 1 sigs from 'big_genomes/25.fa.sig'g'
loaded 1 sigs from 'big_genomes/26.fa.sig'20 sigs total
loaded 1 sigs from 'big_genomes/27.fa.sig'g'
loaded 1 sigs from 'big_genomes/28.fa.sig'g'
loaded 1 sigs from 'big_genomes/29.fa.sig'g'
loaded 1 sigs from 'big_genomes/3.fa.sig'g'
loaded 1 sigs from 'big_genomes/30.fa.sig'g'
loaded 1 sigs from 'big_genomes/31.fa.sig'g'
loaded 1 sigs from 'big_genomes/32.fa.sig'g'
loaded 1 sigs from 'big_genomes/33.fa.sig'g'
loaded 1 sigs from 'big_genomes/34.fa.sig'g'
loaded 1 sigs from 'big_genomes/35.fa.sig'30 sigs total
loaded 1 sigs from 'big_genomes/36.fa.sig'g'
loaded 1 sigs from 'big_genomes/37.fa.sig'g'
loaded 1 sigs from 'big_genomes/38.fa.sig'g'
loaded 1 sigs from 'big_genomes/39.fa.sig'g'
loaded 1 sigs from 'big_genomes/4.fa.sig'g'
loaded 1 sigs from 'big_genomes/40.fa.sig'g'
loaded 1 sigs from 'big_genomes/41.fa.sig'g'
loaded 1 sigs from 'big_genomes/42.fa.sig'g'
loaded 1 sigs from 'big_genomes/43.fa.sig'g'
loaded 1 sigs from 'big_genomes/44.fa.sig'40 sigs total
loaded 1 sigs from 'big_genomes/45.fa.sig'g'
loaded 1 sigs from 'big_genomes/46.fa.sig'g'
loaded 1 sigs from 'big_genomes/47.fa.sig'g'
loaded 1 sigs from 'big_genomes/48.fa.sig'g'
loaded 1 sigs from 'big_genomes/49.fa.sig'g'
loaded 1 sigs from 'big_genomes/5.fa.sig'g'
loaded 1 sigs from 'big_genomes/50.fa.sig'g'
loaded 1 sigs from 'big_genomes/51.fa.sig'g'
loaded 1 sigs from 'big_genomes/52.fa.sig'g'
loaded 1 sigs from 'big_genomes/53.fa.sig'50 sigs total
loaded 1 sigs from 'big_genomes/54.fa.sig'g'
loaded 1 sigs from 'big_genomes/55.fa.sig'g'
loaded 1 sigs from 'big_genomes/56.fa.sig'g'
loaded 1 sigs from 'big_genomes/57.fa.sig'g'
loaded 1 sigs from 'big_genomes/58.fa.sig'g'
loaded 1 sigs from 'big_genomes/59.fa.sig'g'
loaded 1 sigs from 'big_genomes/6.fa.sig'g'
loaded 1 sigs from 'big_genomes/60.fa.sig'g'
loaded 1 sigs from 'big_genomes/61.fa.sig'g'
loaded 1 sigs from 'big_genomes/62.fa.sig'60 sigs total
loaded 1 sigs from 'big_genomes/63.fa.sig'g'
loaded 1 sigs from 'big_genomes/7.fa.sig'g'
loaded 1 sigs from 'big_genomes/8.fa.sig'g'
loaded 1 sigs from 'big_genomes/9.fa.sig'g'
loaded 64 signatures total.

min similarity in matrix: 0.000
saving labels to: compare_all.mat.labels.txt
saving comparison matrix to: compare_all.mat

== This is sourmash version 4.0.0a4.dev12+g31c5eda2. ==
== Please cite Brown and Irber (2016), doi:10.21105/joss.00027. ==

loading comparison matrix from compare_all.mat...
...got 64 x 64 matrix.
loading labels from compare_all.mat.labels.txt
saving histogram of matrix values => compare_all.mat.hist.png
wrote dendrogram to: compare_all.mat.dendro.png
wrote numpy distance matrix to: compare_all.mat.matrix.png
[4]:
from IPython.display import Image
Image(filename='compare_all.mat.matrix.png')
[4]:
_images/sourmash-collections_7_0.png

make a fast(er) search database for all of them

[5]:
!sourmash index -k 31 all-genomes big_genomes/*.sig

== This is sourmash version 4.0.0a4.dev12+g31c5eda2. ==
== Please cite Brown and Irber (2016), doi:10.21105/joss.00027. ==

loading 64 files into SBT
loaded 1 sigs from 'big_genomes/0.fa.sig'g'
loaded 1 sigs from 'big_genomes/1.fa.sig'g'
loaded 1 sigs from 'big_genomes/10.fa.sig'g'
loaded 1 sigs from 'big_genomes/11.fa.sig'g'
loaded 1 sigs from 'big_genomes/12.fa.sig'g'
loaded 1 sigs from 'big_genomes/13.fa.sig'g'
loaded 1 sigs from 'big_genomes/14.fa.sig'g'
loaded 1 sigs from 'big_genomes/15.fa.sig'g'
loaded 1 sigs from 'big_genomes/16.fa.sig'g'
loaded 1 sigs from 'big_genomes/17.fa.sig'10 sigs total
loaded 1 sigs from 'big_genomes/18.fa.sig'g'
loaded 1 sigs from 'big_genomes/19.fa.sig'g'
loaded 1 sigs from 'big_genomes/2.fa.sig'g'
loaded 1 sigs from 'big_genomes/20.fa.sig'g'
loaded 1 sigs from 'big_genomes/21.fa.sig'g'
loaded 1 sigs from 'big_genomes/22.fa.sig'g'
loaded 1 sigs from 'big_genomes/23.fa.sig'g'
loaded 1 sigs from 'big_genomes/24.fa.sig'g'
loaded 1 sigs from 'big_genomes/25.fa.sig'g'
loaded 1 sigs from 'big_genomes/26.fa.sig'20 sigs total
loaded 1 sigs from 'big_genomes/27.fa.sig'g'
loaded 1 sigs from 'big_genomes/28.fa.sig'g'
loaded 1 sigs from 'big_genomes/29.fa.sig'g'
loaded 1 sigs from 'big_genomes/3.fa.sig'g'
loaded 1 sigs from 'big_genomes/30.fa.sig'g'
loaded 1 sigs from 'big_genomes/31.fa.sig'g'
loaded 1 sigs from 'big_genomes/32.fa.sig'g'
loaded 1 sigs from 'big_genomes/33.fa.sig'g'
loaded 1 sigs from 'big_genomes/34.fa.sig'g'
loaded 1 sigs from 'big_genomes/35.fa.sig'30 sigs total
loaded 1 sigs from 'big_genomes/36.fa.sig'g'
loaded 1 sigs from 'big_genomes/37.fa.sig'g'
loaded 1 sigs from 'big_genomes/38.fa.sig'g'
loaded 1 sigs from 'big_genomes/39.fa.sig'g'
loaded 1 sigs from 'big_genomes/4.fa.sig'g'
loaded 1 sigs from 'big_genomes/40.fa.sig'g'
loaded 1 sigs from 'big_genomes/41.fa.sig'g'
loaded 1 sigs from 'big_genomes/42.fa.sig'g'
loaded 1 sigs from 'big_genomes/43.fa.sig'g'
loaded 1 sigs from 'big_genomes/44.fa.sig'40 sigs total
loaded 1 sigs from 'big_genomes/45.fa.sig'g'
loaded 1 sigs from 'big_genomes/46.fa.sig'g'
loaded 1 sigs from 'big_genomes/47.fa.sig'g'
loaded 1 sigs from 'big_genomes/48.fa.sig'g'
loaded 1 sigs from 'big_genomes/49.fa.sig'g'
loaded 1 sigs from 'big_genomes/5.fa.sig'g'
loaded 1 sigs from 'big_genomes/50.fa.sig'g'
loaded 1 sigs from 'big_genomes/51.fa.sig'g'
loaded 1 sigs from 'big_genomes/52.fa.sig'g'
loaded 1 sigs from 'big_genomes/53.fa.sig'50 sigs total
loaded 1 sigs from 'big_genomes/54.fa.sig'g'
loaded 1 sigs from 'big_genomes/55.fa.sig'g'
loaded 1 sigs from 'big_genomes/56.fa.sig'g'
loaded 1 sigs from 'big_genomes/57.fa.sig'g'
loaded 1 sigs from 'big_genomes/58.fa.sig'g'
loaded 1 sigs from 'big_genomes/59.fa.sig'g'
loaded 1 sigs from 'big_genomes/6.fa.sig'g'
loaded 1 sigs from 'big_genomes/60.fa.sig'g'
loaded 1 sigs from 'big_genomes/61.fa.sig'g'
loaded 1 sigs from 'big_genomes/62.fa.sig'60 sigs total
loaded 1 sigs from 'big_genomes/63.fa.sig'g'
loaded 1 sigs from 'big_genomes/7.fa.sig'g'
loaded 1 sigs from 'big_genomes/8.fa.sig'g'
loaded 1 sigs from 'big_genomes/9.fa.sig'g'

loaded 64 sigs; saving SBT under "all-genomes"
Finished saving nodes, now saving SBT index file.
Finished saving SBT index, available at /Users/t/dev/sourmash/doc/all-genomes.sbt.zip

You can now use this to search, and gather.

[6]:
!sourmash search shew_os185.fa.sig all-genomes --threshold=0.001


== This is sourmash version 4.0.0a4.dev12+g31c5eda2. ==

== Please cite Brown and Irber (2016), doi:10.21105/joss.00027. ==

Cannot open file ‘shew_os185.fa.sig’

</pre>

== Please cite Brown and Irber (2016), doi:10.21105/joss.00027. ==

Cannot open file ‘shew_os185.fa.sig’

end{sphinxVerbatim}



== This is sourmash version 4.0.0a4.dev12+g31c5eda2. ==

== Please cite Brown and Irber (2016), doi:10.21105/joss.00027. ==

Cannot open file ‘shew_os185.fa.sig’

[7]:
# (make fake metagenome again, just in case)
!cat genomes/*.fa > fake-metagenome.fa
!rm -f fake-metagenome.fa.sig
!sourmash sketch dna -p k=31,scaled=1000 fake-metagenome.fa

== This is sourmash version 4.0.0a4.dev12+g31c5eda2. ==
== Please cite Brown and Irber (2016), doi:10.21105/joss.00027. ==

computing signatures for files: fake-metagenome.fa
Computing a total of 1 signature(s).
... reading sequences from fake-metagenome.fa
calculated 1 signatures for 3 sequences in fake-metagenome.fa
saved signature(s) to fake-metagenome.fa.sig. Note: signature license is CC0.
[8]:
!sourmash gather fake-metagenome.fa.sig all-genomes

== This is sourmash version 4.0.0a4.dev12+g31c5eda2. ==
== Please cite Brown and Irber (2016), doi:10.21105/joss.00027. ==

select query k=31 automatically.
loaded query: fake-metagenome.fa... (k=31, DNA)
loaded 1 databases.


overlap     p_query p_match
---------   ------- -------
0.5 Mbp       42.2%   10.5%    NC_011663.1 Shewanella baltica OS223,...
499.0 kbp     38.4%   18.5%    CP001071.1 Akkermansia muciniphila AT...
0.5 Mbp       19.4%    4.9%    NC_009665.1 Shewanella baltica OS185,...

found 3 matches total;
the recovered matches hit 100.0% of the query

build a database with taxonomic information –

for this, we need to provide a metadata file that contains accession => tax information.

[9]:
import pandas
df = pandas.read_csv('podar-lineage.csv')
df
[9]:
accession taxid superkingdom phylum class order family genus species strain
0 AE000782 224325 Archaea Euryarchaeota Archaeoglobi Archaeoglobales Archaeoglobaceae Archaeoglobus Archaeoglobus fulgidus Archaeoglobus fulgidus DSM 4304
1 NC_000909 243232 Archaea Euryarchaeota Methanococci Methanococcales Methanocaldococcaceae Methanocaldococcus Methanocaldococcus jannaschii Methanocaldococcus jannaschii DSM 2661
2 NC_003272 103690 Bacteria Cyanobacteria NaN Nostocales Nostocaceae Nostoc Nostoc sp. PCC 7120 NaN
3 AE009441 178306 Archaea Crenarchaeota Thermoprotei Thermoproteales Thermoproteaceae Pyrobaculum Pyrobaculum aerophilum Pyrobaculum aerophilum str. IM2
4 AE009950 186497 Archaea Euryarchaeota Thermococci Thermococcales Thermococcaceae Pyrococcus Pyrococcus furiosus Pyrococcus furiosus DSM 3638
... ... ... ... ... ... ... ... ... ... ...
59 NZ_ABZS01000228 432331 Bacteria Aquificae Aquificae Aquificales Hydrogenothermaceae Sulfurihydrogenibium Sulfurihydrogenibium yellowstonense Sulfurihydrogenibium yellowstonense SS-5
60 NZ_JGWU01000001 1458259 Bacteria Proteobacteria Betaproteobacteria Burkholderiales Alcaligenaceae Bordetella Bordetella bronchiseptica Bordetella bronchiseptica D989
61 NZ_FWDH01000003 31899 Bacteria Firmicutes Clostridia Thermoanaerobacterales Thermoanaerobacterales Family III. Incertae Sedis Caldicellulosiruptor Caldicellulosiruptor bescii NaN
62 NC_009972 316274 Bacteria Chloroflexi Chloroflexia Herpetosiphonales Herpetosiphonaceae Herpetosiphon Herpetosiphon aurantiacus Herpetosiphon aurantiacus DSM 785
63 NC_005213 228908 Archaea Nanoarchaeota NaN Nanoarchaeales Nanoarchaeaceae Nanoarchaeum Nanoarchaeum equitans Nanoarchaeum equitans Kin4-M

64 rows × 10 columns

[10]:
!sourmash lca index podar-lineage.csv taxdb big_genomes/*.sig  -C 3 --split-identifiers

== This is sourmash version 4.0.0a4.dev12+g31c5eda2. ==
== Please cite Brown and Irber (2016), doi:10.21105/joss.00027. ==

Building LCA database with ksize=31 scaled=10000 moltype=DNA.
examining spreadsheet headers...
** assuming column 'accession' is identifiers in spreadsheet
64 distinct identities in spreadsheet out of 64 rows.
64 distinct lineages in spreadsheet out of 64 rows.
... loaded 64 signatures.H01000003.1 Caldicellulo (64 of 64); skipped 0 so far
loaded 19993 hashes at ksize=31 scaled=10000
64 assigned lineages out of 64 distinct lineages in spreadsheet.
64 identifiers used out of 64 distinct identifiers in spreadsheet.
saving to LCA DB: taxdb.lca.json

This database ‘taxdb.lca.json’ can be used for search and gather as above:

[11]:
!sourmash gather fake-metagenome.fa.sig taxdb.lca.json


== This is sourmash version 4.0.0a4.dev12+g31c5eda2. ==

== Please cite Brown and Irber (2016), doi:10.21105/joss.00027. ==

select query k=31 automatically.

loaded query: fake-metagenome.fa… (k=31, DNA)

loading from taxdb.lca.json…

loaded LCA taxdb.lca.json

loaded 1 databases.

overlap p_query p_match ——— ——- ——- 0.6 Mbp 46.7% 11.6% NC_011663.1 Shewanella baltica OS223,… 0.5 Mbp 38.7% 19.3% CP001071.1 Akkermansia muciniphila AT… 0.5 Mbp 14.6% 3.9% NC_009665.1 Shewanella baltica OS185,…

found 3 matches total; the recovered matches hit 100.0% of the query

</pre>

== Please cite Brown and Irber (2016), doi:10.21105/joss.00027. ==

select query k=31 automatically.

loaded query: fake-metagenome.fa{ldots} (k=31, DNA)

loading from taxdb.lca.json{ldots}

loaded LCA taxdb.lca.json

loaded 1 databases.

overlap p_query p_match ——— ——- ——- 0.6 Mbp 46.7% 11.6% NC_011663.1 Shewanella baltica OS223,{ldots} 0.5 Mbp 38.7% 19.3% CP001071.1 Akkermansia muciniphila AT{ldots} 0.5 Mbp 14.6% 3.9% NC_009665.1 Shewanella baltica OS185,{ldots}

found 3 matches total; the recovered matches hit 100.0% of the query

end{sphinxVerbatim}



== This is sourmash version 4.0.0a4.dev12+g31c5eda2. ==

== Please cite Brown and Irber (2016), doi:10.21105/joss.00027. ==

select query k=31 automatically.

loaded query: fake-metagenome.fa… (k=31, DNA)

loading from taxdb.lca.json…

loaded LCA taxdb.lca.json



loaded 1 databases.

overlap p_query p_match ——— ——- ——- 0.6 Mbp 46.7% 11.6% NC_011663.1 Shewanella baltica OS223,… 0.5 Mbp 38.7% 19.3% CP001071.1 Akkermansia muciniphila AT… 0.5 Mbp 14.6% 3.9% NC_009665.1 Shewanella baltica OS185,…

found 3 matches total; the recovered matches hit 100.0% of the query

…but can also be used for taxonomic summarization:

[12]:
!sourmash lca summarize --query fake-metagenome.fa.sig --db taxdb.lca.json


== This is sourmash version 4.0.0a4.dev12+g31c5eda2. ==

== Please cite Brown and Irber (2016), doi:10.21105/joss.00027. ==

… loading database taxdb.lca.json

loaded 1 LCA databases. ksize=31, scaled=10000 moltype=DNA

finding query signatures…

… loading fake-metagenome.fa (file 1 of 1) 38.7% 53 Bacteria;Verrucomicrobia;Verrucomicrobiae;Verrucomicrobiales;Akkermansiaceae;Akkermansia;Akkermansia muciniphila;Akkermansia muciniphila ATCC BAA-835 fake-metagenome.fa.sig:4e1ac0cf fake-metagenome.fa

38.7% 53 Bacteria;Verrucomicrobia;Verrucomicrobiae;Verrucomicrobiales;Akkermansiaceae;Akkermansia;Akkermansia muciniphila fake-metagenome.fa.sig:4e1ac0cf fake-metagenome.fa 38.7% 53 Bacteria;Verrucomicrobia;Verrucomicrobiae;Verrucomicrobiales;Akkermansiaceae;Akkermansia fake-metagenome.fa.sig:4e1ac0cf fake-metagenome.fa 38.7% 53 Bacteria;Verrucomicrobia;Verrucomicrobiae;Verrucomicrobiales;Akkermansiaceae fake-metagenome.fa.sig:4e1ac0cf fake-metagenome.fa 38.7% 53 Bacteria;Verrucomicrobia;Verrucomicrobiae;Verrucomicrobiales fake-metagenome.fa.sig:4e1ac0cf fake-metagenome.fa 38.7% 53 Bacteria;Verrucomicrobia;Verrucomicrobiae fake-metagenome.fa.sig:4e1ac0cf fake-metagenome.fa 38.7% 53 Bacteria;Verrucomicrobia fake-metagenome.fa.sig:4e1ac0cf fake-metagenome.fa 100.0% 137 Bacteria fake-metagenome.fa.sig:4e1ac0cf fake-metagenome.fa 61.3% 84 Bacteria;Proteobacteria;Gammaproteobacteria;Alteromonadales;Shewanellaceae;Shewanella;Shewanella baltica fake-metagenome.fa.sig:4e1ac0cf fake-metagenome.fa 61.3% 84 Bacteria;Proteobacteria;Gammaproteobacteria;Alteromonadales;Shewanellaceae;Shewanella fake-metagenome.fa.sig:4e1ac0cf fake-metagenome.fa 61.3% 84 Bacteria;Proteobacteria;Gammaproteobacteria;Alteromonadales;Shewanellaceae fake-metagenome.fa.sig:4e1ac0cf fake-metagenome.fa 61.3% 84 Bacteria;Proteobacteria;Gammaproteobacteria;Alteromonadales fake-metagenome.fa.sig:4e1ac0cf fake-metagenome.fa 61.3% 84 Bacteria;Proteobacteria;Gammaproteobacteria fake-metagenome.fa.sig:4e1ac0cf fake-metagenome.fa 61.3% 84 Bacteria;Proteobacteria fake-metagenome.fa.sig:4e1ac0cf fake-metagenome.fa 22.6% 31 Bacteria;Proteobacteria;Gammaproteobacteria;Alteromonadales;Shewanellaceae;Shewanella;Shewanella baltica;Shewanella baltica OS223 fake-metagenome.fa.sig:4e1ac0cf fake-metagenome.fa 14.6% 20 Bacteria;Proteobacteria;Gammaproteobacteria;Alteromonadales;Shewanellaceae;Shewanella;Shewanella baltica;Shewanella baltica OS185 fake-metagenome.fa.sig:4e1ac0cf fake-metagenome.fa

loaded 1 signatures from 1 files total.

</pre>

== Please cite Brown and Irber (2016), doi:10.21105/joss.00027. ==

{ldots} loading database taxdb.lca.json

loaded 1 LCA databases. ksize=31, scaled=10000 moltype=DNA

finding query signatures{ldots}

{ldots} loading fake-metagenome.fa (file 1 of 1) 38.7% 53 Bacteria;Verrucomicrobia;Verrucomicrobiae;Verrucomicrobiales;Akkermansiaceae;Akkermansia;Akkermansia muciniphila;Akkermansia muciniphila ATCC BAA-835 fake-metagenome.fa.sig:4e1ac0cf fake-metagenome.fa

38.7% 53 Bacteria;Verrucomicrobia;Verrucomicrobiae;Verrucomicrobiales;Akkermansiaceae;Akkermansia;Akkermansia muciniphila fake-metagenome.fa.sig:4e1ac0cf fake-metagenome.fa 38.7% 53 Bacteria;Verrucomicrobia;Verrucomicrobiae;Verrucomicrobiales;Akkermansiaceae;Akkermansia fake-metagenome.fa.sig:4e1ac0cf fake-metagenome.fa 38.7% 53 Bacteria;Verrucomicrobia;Verrucomicrobiae;Verrucomicrobiales;Akkermansiaceae fake-metagenome.fa.sig:4e1ac0cf fake-metagenome.fa 38.7% 53 Bacteria;Verrucomicrobia;Verrucomicrobiae;Verrucomicrobiales fake-metagenome.fa.sig:4e1ac0cf fake-metagenome.fa 38.7% 53 Bacteria;Verrucomicrobia;Verrucomicrobiae fake-metagenome.fa.sig:4e1ac0cf fake-metagenome.fa 38.7% 53 Bacteria;Verrucomicrobia fake-metagenome.fa.sig:4e1ac0cf fake-metagenome.fa 100.0% 137 Bacteria fake-metagenome.fa.sig:4e1ac0cf fake-metagenome.fa 61.3% 84 Bacteria;Proteobacteria;Gammaproteobacteria;Alteromonadales;Shewanellaceae;Shewanella;Shewanella baltica fake-metagenome.fa.sig:4e1ac0cf fake-metagenome.fa 61.3% 84 Bacteria;Proteobacteria;Gammaproteobacteria;Alteromonadales;Shewanellaceae;Shewanella fake-metagenome.fa.sig:4e1ac0cf fake-metagenome.fa 61.3% 84 Bacteria;Proteobacteria;Gammaproteobacteria;Alteromonadales;Shewanellaceae fake-metagenome.fa.sig:4e1ac0cf fake-metagenome.fa 61.3% 84 Bacteria;Proteobacteria;Gammaproteobacteria;Alteromonadales fake-metagenome.fa.sig:4e1ac0cf fake-metagenome.fa 61.3% 84 Bacteria;Proteobacteria;Gammaproteobacteria fake-metagenome.fa.sig:4e1ac0cf fake-metagenome.fa 61.3% 84 Bacteria;Proteobacteria fake-metagenome.fa.sig:4e1ac0cf fake-metagenome.fa 22.6% 31 Bacteria;Proteobacteria;Gammaproteobacteria;Alteromonadales;Shewanellaceae;Shewanella;Shewanella baltica;Shewanella baltica OS223 fake-metagenome.fa.sig:4e1ac0cf fake-metagenome.fa 14.6% 20 Bacteria;Proteobacteria;Gammaproteobacteria;Alteromonadales;Shewanellaceae;Shewanella;Shewanella baltica;Shewanella baltica OS185 fake-metagenome.fa.sig:4e1ac0cf fake-metagenome.fa

loaded 1 signatures from 1 files total.

end{sphinxVerbatim}



== This is sourmash version 4.0.0a4.dev12+g31c5eda2. ==

== Please cite Brown and Irber (2016), doi:10.21105/joss.00027. ==

  … loading database taxdb.lca.json

  loaded 1 LCA databases. ksize=31, scaled=10000 moltype=DNA

finding query signatures…

  … loading fake-metagenome.fa (file 1 of 1) 38.7% 53 Bacteria;Verrucomicrobia;Verrucomicrobiae;Verrucomicrobiales;Akkermansiaceae;Akkermansia;Akkermansia muciniphila;Akkermansia muciniphila ATCC BAA-835 fake-metagenome.fa.sig:4e1ac0cf fake-metagenome.fa

38.7% 53 Bacteria;Verrucomicrobia;Verrucomicrobiae;Verrucomicrobiales;Akkermansiaceae;Akkermansia;Akkermansia muciniphila fake-metagenome.fa.sig:4e1ac0cf fake-metagenome.fa 38.7% 53 Bacteria;Verrucomicrobia;Verrucomicrobiae;Verrucomicrobiales;Akkermansiaceae;Akkermansia fake-metagenome.fa.sig:4e1ac0cf fake-metagenome.fa 38.7% 53 Bacteria;Verrucomicrobia;Verrucomicrobiae;Verrucomicrobiales;Akkermansiaceae fake-metagenome.fa.sig:4e1ac0cf fake-metagenome.fa 38.7% 53 Bacteria;Verrucomicrobia;Verrucomicrobiae;Verrucomicrobiales fake-metagenome.fa.sig:4e1ac0cf fake-metagenome.fa 38.7% 53 Bacteria;Verrucomicrobia;Verrucomicrobiae fake-metagenome.fa.sig:4e1ac0cf fake-metagenome.fa 38.7% 53 Bacteria;Verrucomicrobia fake-metagenome.fa.sig:4e1ac0cf fake-metagenome.fa 100.0% 137 Bacteria fake-metagenome.fa.sig:4e1ac0cf fake-metagenome.fa 61.3% 84 Bacteria;Proteobacteria;Gammaproteobacteria;Alteromonadales;Shewanellaceae;Shewanella;Shewanella baltica fake-metagenome.fa.sig:4e1ac0cf fake-metagenome.fa 61.3% 84 Bacteria;Proteobacteria;Gammaproteobacteria;Alteromonadales;Shewanellaceae;Shewanella fake-metagenome.fa.sig:4e1ac0cf fake-metagenome.fa 61.3% 84 Bacteria;Proteobacteria;Gammaproteobacteria;Alteromonadales;Shewanellaceae fake-metagenome.fa.sig:4e1ac0cf fake-metagenome.fa 61.3% 84 Bacteria;Proteobacteria;Gammaproteobacteria;Alteromonadales fake-metagenome.fa.sig:4e1ac0cf fake-metagenome.fa 61.3% 84 Bacteria;Proteobacteria;Gammaproteobacteria fake-metagenome.fa.sig:4e1ac0cf fake-metagenome.fa 61.3% 84 Bacteria;Proteobacteria fake-metagenome.fa.sig:4e1ac0cf fake-metagenome.fa 22.6% 31 Bacteria;Proteobacteria;Gammaproteobacteria;Alteromonadales;Shewanellaceae;Shewanella;Shewanella baltica;Shewanella baltica OS223 fake-metagenome.fa.sig:4e1ac0cf fake-metagenome.fa 14.6% 20 Bacteria;Proteobacteria;Gammaproteobacteria;Alteromonadales;Shewanellaceae;Shewanella;Shewanella baltica;Shewanella baltica OS185 fake-metagenome.fa.sig:4e1ac0cf fake-metagenome.fa

  loaded 1 signatures from 1 files total.

Other pointers

Sourmash: a practical guide

Classifying signatures taxonomically

Pre-built search databases

A full list of notebooks

An introduction to k-mers for genome comparison and analysis

Some sourmash command line examples!

Working with private collections of signatures.

Using the LCA_Database API.

[ ]: