Difference between revisions of "Catalogue"

From Dynamo
Jump to navigation Jump to search
 
(23 intermediate revisions by the same user not shown)
Line 5: Line 5:
 
If you have already a set of subtomograms, you have already all the initial orientations you could reasonably extract from the tomograms and you don't want to extract the tomograms again, you probably don't need to define a catalogue. You already have all you need to start designing alignment projects.
 
If you have already a set of subtomograms, you have already all the initial orientations you could reasonably extract from the tomograms and you don't want to extract the tomograms again, you probably don't need to define a catalogue. You already have all you need to start designing alignment projects.
  
Keeping a ''catalogue'' is not necessary for cropping particles out of your tomograms. The tools associated with the catalogue can easily be used independently at any stage. We however encourage you to  organize your tomograms in a catalogue from the very beginning, as this induces a very smooth workflow and eliminates lengthy administrative overheads that are prone to happen on a later stage.
+
Keeping a ''catalogue'' is not necessary for [[particle extraction|cropping particles]] out of your tomograms. The tools associated with the catalogue can easily be used independently at any stage. We however encourage you to  organize your tomograms in a catalogue from the very beginning, as this induces a very smooth workflow and eliminates lengthy administrative overheads that are prone to happen on a later stage.
 +
 
 +
The main tool to work with catalogues is the [[dcm GUI|<tt>dcm</tt> GUI]], that also provides command line functionality to create, edit an extract information from catalogues. 
 +
Help on command syntax: {{docfunction|dynamo_catalogue_manager|dcm}}
 +
 
 +
==Getting started==
 +
The best way to get familiar with the logic of the ''Dynamo'' catalogues is by [[dynamo_catalogue_tutorial|generating a test catalogue]] for a set of automatically created toy tomograms.
 +
 
 +
If you already have a data set of several tilt series, you can check our [[Tips for management of tomographic data sets|general guidelines for organization of tomography projects]]. These guidelines sketch a comfortable way to organize your raw data and create your first catalogue.
 +
 
 +
== Basic functionalities ==
 +
 
 +
In the command line, most functionalities are accessible trough the <tt>dcm</tt> command. Type <tt>help dcm</tt> for an exhaustive list.
 +
 
 +
Note that if you apply these functionalities on a catalogue currently in scope in the <tt>dcm gui</tt>, the GUI will '''not''' update automatically. You need to press explicitly on the [list volumes] button. If these functionalities are invoked through the GUI, however, the list in the GUI will update automatically.
 +
 
 +
=== Creating a catalogue ===
 +
 
 +
In the command line, you can just type:
 +
 
 +
<tt>dcm -create myCatalogue</tt>
 +
 
 +
It will create a new catalogue in the current folder. It will appear as a folder called <tt>myCatalogue</tt> and a file called <tt>myCatalogue</tt>.
 +
 
 +
=== Entering a set of tomograms ===
 +
 
 +
==== During creation ====
 +
You can enter directly a set of tomograms in the moment of creation of the catalogue through the flag <tt>fromvll</tt>
 +
 
 +
<tt>dcm -create myCatalogue -fromvll myList.vll</tt>
 +
 
 +
Here <tt>myList.vll</tt> is a [[Volume list file | volume list]]. This is just a text file; the syntax can be rather complex, but in its simplest form it can be just a file with a tomogram file name at each line, like:
 +
 
 +
<nowiki>tomogram1.mrc
 +
tomogram2.mrc</nowiki>
 +
 
 +
or you can [[Volume_list_file#Advanced_syntax | pass metadata]] of each tomogram
 +
 
 +
<nowiki>tomogram1.mrc
 +
* def=1.2;
 +
* apix=0.6;
 +
* ftype =1;
 +
* ytilt = -45 60;
 +
 
 +
tomogram2.mrc</nowiki>
 +
 
 +
This metadata can be also passed in a later stage by editing the [[dcm GUI|<tt>dcm</tt> GUI]].
 +
 
 +
==== Into an existing catalogue ====
 +
In this case, the flag is <tt>addvll</tt> and you use the <tt>c</tt> flag to select an already exisitng catalogue (instead of <tt>create</tt>).
 +
 
 +
<tt>dcm -c myCatalogue -addvll myList.vll</tt>
 +
 
 +
=== Entering a single tomogram ===
 +
 
 +
You can use the <tt>addTomogram</tt> (short <tt>at</tt>) flag:
 +
 
 +
<tt>dcm -c myCatalogue -at myTomogram.em</tt>
 +
 
 +
to add a single tomogram to an existing catalogue.
 +
 
 +
== Moving catalogues ==
 +
 
 +
Catalogues and tomograms that are pointed to by a catalogue '''cannot''' in general just moved in your file system l. Some caution is required when you move them across your file system.
 +
 
 +
A catalogue can be copied to a different location in the file system by copying ''both'' <tt>.ctlg</tt> file and corresponding folder. If the tomogram files linked by the catalogue are expressed through their ''absolute path'', you don't need to do anything especial.
 +
If the tomogram files where expressed with relative paths (a discouraged practice), you will need to relink their paths after moving the catalogue.
 +
 
 +
=== Relinking tomogram paths===
 +
 
 +
This situation occurs when you move all your tomograms from one  location into a new one. You can substitute the string  <tt>oldLocation </tt> with <tt>newLocation</tt> in all the paths in your catalogue by:
 +
 
 +
<tt>dcm -c testCatalogue_withmodels -rl  <oldLocation>  -nl <newLocation> </tt>
 +
 
 +
Remember than relinking tomogram paths is '''not''' necessary if you just moved the catalogue  (without moving the tomograms) and the tomogram files were expressed in their absolute names.
 +
 
 +
=== Relinking models ===
 +
 
 +
Models are automatically relinked after a general relinking of tomogram paths with the model synchronization function. Just press on the corresponding button in the GUI, or use the command <tt>dcm</tt> with flag <tt> synchronize </tt>  tuned to the literal value <tt> models </tt>, i.e.
 +
 
 +
<tt>dcm -c myCatalogue -synchronize models </tt>

Latest revision as of 20:28, 14 August 2017


The catalogue is a simple way to keep track of the work that has been done on groups of tomograms. This work normally aims at the definition of positions (and possibly orientations) inside each tomogram, so that subvolumes can be defined and cropped, to produce an unique data folder with its corresponding table that can be feed into a project for alignment or classification.

If you have already a set of subtomograms, you have already all the initial orientations you could reasonably extract from the tomograms and you don't want to extract the tomograms again, you probably don't need to define a catalogue. You already have all you need to start designing alignment projects.

Keeping a catalogue is not necessary for cropping particles out of your tomograms. The tools associated with the catalogue can easily be used independently at any stage. We however encourage you to organize your tomograms in a catalogue from the very beginning, as this induces a very smooth workflow and eliminates lengthy administrative overheads that are prone to happen on a later stage.

The main tool to work with catalogues is the dcm GUI, that also provides command line functionality to create, edit an extract information from catalogues. Help on command syntax: dcm

Getting started

The best way to get familiar with the logic of the Dynamo catalogues is by generating a test catalogue for a set of automatically created toy tomograms.

If you already have a data set of several tilt series, you can check our general guidelines for organization of tomography projects. These guidelines sketch a comfortable way to organize your raw data and create your first catalogue.

Basic functionalities

In the command line, most functionalities are accessible trough the dcm command. Type help dcm for an exhaustive list.

Note that if you apply these functionalities on a catalogue currently in scope in the dcm gui, the GUI will not update automatically. You need to press explicitly on the [list volumes] button. If these functionalities are invoked through the GUI, however, the list in the GUI will update automatically.

Creating a catalogue

In the command line, you can just type:

dcm -create myCatalogue

It will create a new catalogue in the current folder. It will appear as a folder called myCatalogue and a file called myCatalogue.

Entering a set of tomograms

During creation

You can enter directly a set of tomograms in the moment of creation of the catalogue through the flag fromvll

dcm -create myCatalogue -fromvll myList.vll

Here myList.vll is a volume list. This is just a text file; the syntax can be rather complex, but in its simplest form it can be just a file with a tomogram file name at each line, like:

tomogram1.mrc 
tomogram2.mrc

or you can pass metadata of each tomogram

tomogram1.mrc 
* def=1.2;
* apix=0.6;
* ftype =1;
* ytilt = -45 60;

tomogram2.mrc

This metadata can be also passed in a later stage by editing the dcm GUI.

Into an existing catalogue

In this case, the flag is addvll and you use the c flag to select an already exisitng catalogue (instead of create).

dcm -c myCatalogue -addvll myList.vll

Entering a single tomogram

You can use the addTomogram (short at) flag:

dcm -c myCatalogue -at myTomogram.em

to add a single tomogram to an existing catalogue.

Moving catalogues

Catalogues and tomograms that are pointed to by a catalogue cannot in general just moved in your file system l. Some caution is required when you move them across your file system.

A catalogue can be copied to a different location in the file system by copying both .ctlg file and corresponding folder. If the tomogram files linked by the catalogue are expressed through their absolute path, you don't need to do anything especial. If the tomogram files where expressed with relative paths (a discouraged practice), you will need to relink their paths after moving the catalogue.

Relinking tomogram paths

This situation occurs when you move all your tomograms from one location into a new one. You can substitute the string oldLocation with newLocation in all the paths in your catalogue by:

dcm -c testCatalogue_withmodels -rl <oldLocation> -nl <newLocation>

Remember than relinking tomogram paths is not necessary if you just moved the catalogue (without moving the tomograms) and the tomogram files were expressed in their absolute names.

Relinking models

Models are automatically relinked after a general relinking of tomogram paths with the model synchronization function. Just press on the corresponding button in the GUI, or use the command dcm with flag synchronize tuned to the literal value models , i.e.

dcm -c myCatalogue -synchronize models