We recommend running BAli-Phy on a computing cluster for long runs. A computing cluster can speed up the analysis by allowing you to run several identical MCMC chains simultaneously and then pool the resulting samples. You also don't need to worry that logging out or turning off the computer will terminate the run early. Result files can be copied back to a laptop or desktop for viewing.

We typically run BAli-Phy on computers with 8Gb of RAM. You may need a 64-bit executable and a 64-bit version of your operating system to be able to analyze large data sets that consume more that 2Gb of RAM.

Before you can use BAli-Phy on MS Windows, you must first install Cygwin. Cygwin is a Unix/Linux command-line environment for Windows. While running the Cygwin installer setup.exe, you will be given an opportunity to select additional packages.

You may then access the Unix command line environment by running the Cygwin Terminal application (not the normal windows command line).

You might wish to save the installer on your desktop in case you want to run it again, since you can use it to install additional packages later.

You can now install with homebrew if you have the XCode 7 (or higher) compiler. The recipe is in homebrew/science. However, the recipe may not install the latest version.

No extra requirements.

The following software is important to install:

The following software is recommended to install:

GNUplot can be installed using the Cygwin installer on Windows systems. It can also be installed using macports or homebrew on Macintosh systems, but installing these package managers requires first installing Xcode Developer Tools.

In order to determine that the software has been correctly installed, and the PATH has been correctly set, run the following commands:

% bali-phy ~/Applications/bali-phy-3.0-beta2/share/bali-phy/examples/sequences/5S-rRNA/5d.fasta --iter=150
% bp-analyze.pl 5d-1/

Furthermore, the directory 5d-1 should contain a file called C1.log. You should be able to load this file in Tracer, although the chain will not really have converged yet.

In order to compile BAli-Phy, you need a C++ compiler than can understands the C++14 standard.

In order to compile the program on UNIX, first extract the source code archive, using a graphical archive manager, or the command-line tool tar:

% tar -zxf bali-phy-3.0-beta2.tar.gz

Then create a separate build directory, enter it, and run the configure command:

% mkdir build
% cd build
% ../bali-phy-3.0-beta2/configure --prefix=$HOME/Applications/bali-phy-3.0-beta2/

If this command succeeds, then you can simply type

% make
% make install

to build and install bali-phy and its associated tools and install it in ~/Applications/bali-phy-3.0-beta2. To customize the compilation and installation process, read the following sections on supplying arguments to the configure script.

Skip this step unless you are compiling a snapshot of the source code that you checked out using GIT. If you downloaded an official tar.gz archive of the source from the website, then it already includes these files.

To generate these files, you need automake 1.8 (or higher) autoconf 2.59 (or higher), and libtool. In the top level directory of the repository that you checked out, run

% ./bootstrap.sh

If your system has multiple versions of automake, then you may have to type e.g. automake-1.14 -a and aclocal-1.14 instead in order to specify which version to use.

After compiling BAli-Phy, you can simply type make install. This will copy the compiled executables to the installation directory (See Section 3.3.1, “Installing to another location”).

The simplest way to run BAli-Phy is to type all the arguments on the command line:

% bali-phy sequence-file

Here sequence-file is a FastA or PHYLIP file containing the sequences you wish to analyze. The filename should end in .fasta or .phy to indicate which format it is using.

In this simple example, bali-phy automatically detects whether sequence-file contains DNA, RNA, or Amino-Acids and uses default values for several command line options. Thus, if sequence-file contains DNA, then this is equivalent to the more verbose command line

% bali-phy sequence-file --alphabet DNA --smodel TN --imodel RS07 --iterations=100000

Here the substitution model is Tamura-Nei, the insertion/deletion model is RS07, and the number of iterations is 100,000. If sequence-file contains amino acids, then the defaults will be:

% bali-phy sequence-file --alphabet Amino-Acids --smodel LG --imodel RS07 --iterations=100000

You can specify a more complex substitution model as follows (See Section 7.2, “Basic CTMC models”):

% bali-phy sequence-file --smodel LG+GammaRates+INV

You may specify an indel model of none to fix the alignment to its initial value, and ignore information in shared insertions or deletions.

% bali-phy sequence-file --imodel none

You may analyze multiple genes by putting each one it its own data partition:

% bali-phy sequence-file1 sequence-file2

You should put the data from the first gene in sequence-file1 and the second gene in sequence-file2. The sequence names in both files should be the same. In this scenario, both genes share the same tree, but their alignments vary independently. Furthermore, the branch lengths for each gene are scaled by an independent factor. By default, each partition will have its own default alphabet, substitution model, insertion/deletion model, and tree length.

By default, each partition will recieve an independent copy of the model, and will not share parameter values:

% bali-phy sequence-file1 sequence-file2 --smodel TN --imodel RS07

However, you can select partition-specific values for 5 options: --smodel, --imodel, --alphabet, and --scale. For example, to specify different substitution models but the same alphabet:

% bali-phy sequence-file1 sequence-file2 --smodel 1:TN --smodel 2:GTR --alphabet 1,2:DNA

You can fix the alignment and ignore insertion/deletion information in one partition, while allowing the alignment to vary and using insertion/deletion information in another partition:

% bali-phy sequence-file1 sequence-file2 --imodel 1:RS07 --imodel 2:none

You can also specify that two partitions share a single copy of a single substitution model or indel model. This reduces the number of parameters and also pools information between the partitions:

% bali-phy sequence-file1 sequence-file2 --smodel 1,2:TN --imodel 1,2:RS07

By default each partition has a separate scale, but you can force groups of partitions to share a scale. If you leave the value of the scale blank, the default distribution on scales will be used:

% bali-phy sequence-file1 sequence-file2 --smodel 1:TN --smodel 2:GTR --scale 1,2:

Finally, you may specify -Inone or --imodel none, which affects all partitions:

% bali-phy sequence-file1 sequence-file2 --smodel 1:TN --smodel 2:GTR -t

Running bali-phy on a computing cluster is not necessary, but can speed up the analysis dramatically. This is because a cluster allows you to run several independent MCMC chains simultaneously and pool the resulting samples. You can run multiple chains simultaneously simply by starting several different instances of bali-phy. Each instance of bali-phy runs only one chain and does not require using MPI or special command-line options.

This approach to parallel computation is sometimes more efficient than MCMCMC-based parallelism involving heated chains. It is equivalent to running MCMCMC with no temperature difference between chains, with the exception that it allows results from all chains to be used, instead of just results from the single "cold" chain. Thus, if you run 10 independent chains in parallel, then you may gather samples 10 times faster that a single chain.

In addition to using the command line, you may also specify options in a file. Using an option file can be more convenient if you are going to run the same analysis many times, or if the number of options is large. Furthermore, the option file may contain comments and blank lines. Option files are a good to record what options you used in an analysis, and why.

An option file is specified with the command line option --config file or -cfile. If values for an option are given both on the command line and in an option file, then the command line value overrides the value in the option file.

#select a data set to analyze
align = examples/sequences/EF-Tu/5d.fasta

#select an substitution model
smodel = LG+log_normal_rates+INV

#fix the alignment and do not model indels
imodel = none
	  

% bali-phy examples/sequences/EF-Tu/5d.fasta --smodel LG+log_normal_rates+INV --imodel none

Here are some examples which demonstrate how to run BAli-Phy. In order to run these examples, you must find the examples/sequences/ directory which contains the example files. If you downloaded executables and extracted them in the ~/Applications directory, then the examples/sequences/ directory will be found at ~/Applications/bali-phy-3.0-beta2/share/bali-phy/examples/sequences/.

Also note that bali-phy does not run until it is "finished", but continues to gather samples until the user determines that enough samples have been gathered, and stops it. Thus, it is useful to continually examine the output files while the program is running.

 % bali-phy ~/Applications/bali-phy-3.0-beta2/share/bali-phy/examples/sequences/EF-Tu/5d.fasta

 % bali-phy ~/Applications/bali-phy-3.0-beta2/share/bali-phy/examples/sequences/EF-Tu/5d.fasta --smodel LG+log_normal_rates+INV

 % bali-phy ~/Applications/bali-phy-3.0-beta2/share/bali-phy/examples/sequences/5S-rRNA/25-muscle.fasta -Inone

BAli-Phy can read in sequences and alignments in both FastA and PHYLIP formats. Filenames for FastA files should end in .fasta, .mpfa, .fna, .fas, .fsa, or .fa. Filenames for PHYLIP files should end in .phy. If one of these extensions is not used, then BAli-Phy will attempt to guess which format is being used.

Large data sets run more slowly than small data sets. We recommend a conservative starting point with few taxa and short sequence lengths. You can then increase the size of your data set until a balance between speed and size is reached. The tool alignment-thin described in Section 11, “Alignment utilities” can be used to construct a smaller data set.

The number of samples that you need depends on whether you are primarily interested in obtaining a point estimate or in obtaining detailed measures of confidence and uncertainty. For detailed measures of confidence and uncertainty you should obtain a minimum of 10,000 samples after the Markov chain converges. For an estimate, you don't need very many samples after convergence. (But you may need many samples to be sure that you've converged!)

See also Section 4.4, “Running on computing clusters”.

BAli-Phy creates a new directory to store its output files each time it is run. By default, the directory name is the name of the sequence file, with a number added on the end to make it unique. BAli-Phy first checks if there is already a directory called file-1/, and then moves on to file-2/, etc. until it finds an unused directory name.

You can specify a different name to use instead of the sequence-file name by using the --name option.

In order to infer ancestral sequences at internal nodes, add the option --set log-ancestral=1. This will lead to the creation of an additional output file called C1.P1.ancestral.fastas. This file contains a sampled alignment for each iteration, where N's are replaced with a letter randomly sampled from the posterior distribution.

BAli-Phy writes the following output files inside the directory that it creates:

For the last two files, each line in these files corresponds to one iteration.

This section is primarily oriented to extracting estimates from output files. See Section 10, “Convergence and Mixing: Is it done yet?” for methods of determine effective sample sizes, and for checking mixing and convergence.

% trees-consensus C1.trees > c50.PP.tree

% trees-consensus dir-1/C1.trees dir-2/C1.trees > c50.PP.tree

% trees-consensus -s20% C1.trees > c50.PP.tree

% trees-consensus -s20% -x10 C1.trees > c50.PP.tree

% trees-consensus -s20% -x10 --consensus-PP=0.66 C1.trees > c66.PP.tree

% trees-consensus -s20% -x10 C1.trees --consensus-PP=0.66:c66.PP.tree

% trees-consensus -s20% -x10 C1.trees --consensus-PP=0.5:c50.PP.tree,0.66:c66.PP.tree

% trees-consensus -s20% -x10 C1.trees --consensus=0.5:c50.PP.tree,0.66:c66.PP.tree

% trees-consensus --skip=burnin C1.trees --map-tree=MAP.tree

% trees-bootstrap dir-1/C1.trees dir-2/C1.trees

% statreport C1.log > Report 

% statreport dir-1/C1.log dir-2/C1.log > Report 

% cut-range --skip=burn-in < C1.Pp.fastas | alignment-max > Pp-max.fasta

% alignment-find < C1.MAP > P1-MAP.fasta

% cut-range --skip=burn-in < C1.Pp.fastas | alignment-gild alignment.fasta tree  > alignment-AU.prob
% alignment-draw alignment.fasta --AU alignment-AU.prob > alignment-AU.html

Instead of manually running each of the steps to analyze the output files, you may instead run the PERL script bp-analyze.pl to execute these commands. The script will create an HTML page Results/index.html that summarizes the posterior distribution.

You may run bp-analyze.pl inside the output directory, like this:

% bp-analyze.pl --burnin=iterations

You may also run it with one or more output directories as arguments, like this:

% bp-analyze.pl --burnin=iterations directory-1/ directory-2/

In this case, output from multiple runs will be used to assess convergence and mixing, as well as to increase the precision of the estimates.

All the commands that are executed by bp-analyze.pl will be logged to Results/bp-analyze.log. You can also see these commands as they are executed by supplying the --verbose option:

% bp-analyze.pl --burnin=iterations --verbose

If the substitution model is not specified, then the default model for the alphabet is used. For DNA or RNA, the default model is TN. For Triplets, the default is TNx3. For Codons, the default model is M0. For Amino-Acids, the default model is LG.

The basic substitution models in BAli-Phy are continuous-time Markov chains (CTMC). CTMC models can be characterized by transition rates $Q_{\mathrm{ij}}$ from letter $i$ to letter $j$. After a given time $t$ the probability for transition from state $i$ to state $j$ is given by $$P(t{)}_{\mathrm{ij}}=e^{Q_{\mathrm{ij}}\times t}$$ using a matrix exponential. Becase the CTMC models used in BAli-Phy are all reversible, the rate matrix for these reversible models can be decomposed into a symmetric matrix $S$ and equilibrium frequencies $\pi$ as follows: $$Q_{\mathrm{ij}}=S_{\mathrm{ij}}\times {\pi }_j$$ The matrix $S$ is called the exchangability matrix, and represents how exchangeable letters $i$ and $j$ are, independent of their frequencies.

The basic CTMC models are EQU, HKY, TN, GTR, HKYx3, TNx3, GTRx3, JTT, WAG, LG, and M0. Each of these models is a way of specifying the exchangeability matrix $S_{\mathrm{ij}}$.

The rate matrix $Q_{\mathrm{ij}}$ can be more generally expressed as $$Q_{\mathrm{ij}}=S_{\mathrm{ij}}\times \frac{{\pi }_j^f}{{\pi }_i^{1-f}},$$ where $f$ ranges from $0$ to $1$. Here the parameter $f$ specifies the relative importance of unequal conservation ($f=0$) and unequal replacement ($f=1$) in maintaining the equilibrium frequencies $\pi$.

In fact, this can be generalized even further to $$Q_{\mathrm{ij}}=S_{\mathrm{ij}}\times R(\pi {)}_{\mathrm{ij}}$$ where $${\pi }_i\times R_{\mathrm{ij}}={\pi }_j\times R_{\mathrm{ji}}.$$

These models can therefore be expressed as a combination of an "exchange model" (for $S$) and a "frequency model" (for $R$).

Complex substitution models in BAli-Phy are constructed as mixtures of reversible CTMC models (see Section 7.2, “Basic CTMC models”) that run at different rates (e.g. ${\Gamma }_4+\mathrm{INV}$) or have different parameters (e.g. an M2a codon model).

In order to use the branch-site substitution model, the user needs to

Here is an example tree file:

Any branch lengths provided will be used as initial values in the MCMC analysis. However, it is not necessary to provide them:

The NHX attribute must be applied to the branch, not the node. Therefore it must occur after a colon. Multiple branches may be marked as foreground branches.

An example command line is as follows:

% bali-phy alignment.fasta --smodel branch-site[HKY,F3x4] --disable=topology --tree=tree.tree

The posterior probability of positive selection is the posterior mean of the posSelection parameter. This may be computed using the statreport program with the --mean option.

In case this probability is extremely close to 1 or 0, you may wish to add the option --Rao-Blackwellize S1.BranchSiteTest.posSelection. This will report the log-probability of positive selection each iteration. The user may exponentiate the reported values and then average them (using R, for example) in order to compute a more accurate estimate of the posterior probability of positive selection.

Example: --smodel WAG+F+log_normal_rates+INV

Example: --smodel WAG+log_normal_rates+INV (same as above)

Example: --smodel LG+gwF+log_normal_rates+INV

Example: --smodel EQU --alphabet Triplets

Example: --smodel HKY

Example: --smodel M0

Example: --smodel M0+F1x4

Example: --smodel M2a

Example: --smodel M2a[HKY] (same as above)

When using a codon-based substitution model like M0, you may select the genetic code by specifying --alphabet Codons[genetic-code]. Available genetic codes are standard, mt-vert, mt-invert, mt-yeast, mt-protozoan.

If the genetic code is not specified, then the standard code is used:

% bali-phy sequence-file --smodel M0 --alphabet Codons

This example specifies the vertebrate mitochondrial code:

% bali-phy sequence-file --smodel M0 --alphabet Codons[mt-vert]

The first line of the file is a header consisting of an ordered list of sequence names separated by spaces. Each subsequent line consists of a space-separated list of sequence positions, with the first position corresponding to the first leaf sequence, the second position corresponding to the second leaf sequence, etc. Thus, if there are n leaf taxa, then each line corresponds to a space-separated list of n integers.

For example, the file

A B C
1 2 2

implies that position 1 of leaf sequence A is aligned to position 2 of leaf sequences B and C. Note that the first position in a sequence is position 0.

Optionally, one may use a '-' instead of an integer, which denotes a lack of constraint for that sequence. This can be useful as follows:

A B C D
2 2 - - 
- - 2 2
	

The above constraints force alignment between position 2 of sequences A and B, and between position 2 of sequence C and D.

The program alignment-indices may be used to aid in computing a constraint file from an input alignment. See Q: 13.7.3.

Convergence refers to the the tendency of a Markov chain to to "forget" its starting value and become typical of its equilibrium distribution. Note that convergence is a property of the Markov chain itself, not of individual runs of the Markov chain. Ideally a number of individual runs should be examined in order to determine how many initial iterations to discard as "burnin".

In MCMC, each sample is not fully independent of previous samples. In fact, even after a Markov chain has converged, it can get "stuck" in one part of the parameter space for a long time, before jumping to an equally important part. When this happens, each new sample contributes very little new information, and we need to obtain many more samples to get good precision on our parameter estimates. In such a case, we say that the chain isn't "mixing" well.

Potential Scale Reduction Factors check that different runs have similar posterior distributions. Only numerical variables may have a PSRF. To calculate the PSRF for each numerical parameter, you may run:

% statreport dir-1/C1.log dir-2/C2.p ... dir-n/C1.log > Report 

The PSRF is a ratio of the width of the pooled distribution to the average width of each distribution, and should ideally be 1. The PSRF is customarily considered to be small enough if it is less than 1.01.

We compare the PSRF based on the length of 80% credible intervals (Brooks and Gelman 1998) and report the result as PSRF-80%CI. For integer-valued parameters, we avoid excessively large PSRF values by subtracting 1 from the width of the pooled CI.

We also report a new PSRF that is more sensitive for integer distributions. For each individual distribution, we find the 80% credible interval. We divide the probability of that interval (which may be more than 80%) by the probability of the same interval under the pooled distribution. The average of this measure over all distributions gives us a PSRF that we report as PSRF-RCF.

This convergence diagnostic gives a criterion for detecting when a parameter value has stabilized at different values in several independent runs, indicating a lack of convergence. This situation might occur if different runs of the Markov chain were trapped in different modes and failed to adequately mix between modes.

Show basic information about the alignment:

% alignment-info file.fasta
% alignment-info file.fasta file.tree

To select columns from an alignment:

% alignment-cat -c1-10,50-100,600- file.fasta > result.fasta
% alignment-cat -c5-250/3 file.fasta > first_codon_position.fasta
% alignment-cat -c6-250/3 file.fasta > second_codon_position.fasta

To concatenate two or more alignments:

% alignment-cat file1.fasta file2.fasta > all.fasta

Remove columns without a minimum number of letters:

% alignment-thin --min-letters=5 file.fasta > file-thinned.fasta

Remove sequences:

% alignment-thin --remove=seq1,seq2 file.fasta > file2.fasta

Remove short sequences:

% alignment-thin --longer-than=250 file.fasta > file-long.fasta

Remove sequences while preserving sequence diversity:

% alignment-thin --down-to=30 file.fasta > file-30taxa.fasta
% alignment-thin --down-to=30 file.fasta --keep=seq1,seq2 > file-30taxa.fasta

Remove sequences that are missing conserved columns:

% alignment-thin --remove-crazy=10 file.fasta > file2.fasta

Draw an alignment to HTML, optionally coloring residues by AU.

%  alignment-draw file.fasta --show-ruler --color-scheme=DNA+contrast > file.html
%  alignment-draw file.fasta --show-ruler --AU=file-AU.prob --color-scheme=DNA+contrast+fade+fade+fade+fade > file-AU.html

Find the last (or first) FastA alignment in a file.

% alignment-find --first < file.fastas > first.fasta
% alignment-find < file.fastas > last.fasta

Turn columns from a template alignment into alignment constraints:

% alignment-indices template.fasta > constraints.txt
% alignment-indices -c100-110,200,300- template.fasta > constraints.txt

Each line in this file corresponds to one alignment column.

Remove internal-node ancestral sequences from an alignment. (This probably only works for alignments output by bali-phy.)

% alignment-chop-internal file.fasta > file-chopped.fasta

This program analyzes the tree sample contained in file. It reports the MAP topology, the supported taxa partitions (including partial partitions), and the majority consensus topology.

Usage: trees-bootstrap file1 [file2 ... ] --predicates predicate-file [OPTIONS]

This program analyzes the tree samples contained in file1, file2, etc. It gives the support of each tree sample for each predicate in predicate-file, and reports a confidence interval based on the block bootstrap.

Each predicate is the intersection of a set of partitions, and is specified as a list of partitions or (multifurcating) trees, one per line. Predicates are separated by blank lines.

Usage: trees-to-SRQ predicate-file [OPTIONS] trees-file

This program analyzes the tree samples contained in trees-file. It uses them to produce an SRQ plot for each predicate in predicate-file. Plots are produced in gnuplot format, with one point per line and with plots separated by a blank line.

If --mode sum is specified, then a "sum" plot is produced instead of an SRQ plot. In this plot, the slope of the curve corresponds to the posterior probability of the event. If the --invert option is used then the slope of the curve correspond to the probability of the inverse event. This is recommended if the probability of the event is near 1.0, because the sum plot does not distinguish variation in probabilities near 1.0 well.