Friday 30 June 2017

Criteria for Grouping People into Y-DNA Genetic Families

One of the main tasks of Surname Project administrators is to place new members into the appropriate genetic group within their surname project.

Having run a variety of surname projects for the last few years, I have come up with a set of criteria I use on a routine basis to place newcomers into existing genetic groups and also to identify new genetic groups. I call these criteria Markers of Potential Relatedness (MPRs). And (not surprisingly) these can be thought of as indicators that two people may be "related" to each other, which for the purposes of surname projects means somewhere in the last 1000 years or so. This arbitrary timepoint is chosen because many European surnames were introduced about 1000 years ago (in particular British and Irish surnames), although they only became commonplace several centuries thereafter.

This approach to grouping works best with hereditary surnames (i.e. passed from father to son) but should also work with patronymic (and other) surnames, except that (in these latter cases) criteria 1 and 8 will not apply. The discussion below is very much from the standpoint of hereditary surname projects.

Not all criteria have to be met. But the more criteria that are met, the higher the likelihood of two people being related. This is particularly important in relation to SDSs (Surname or DNA Switches; also known as NPEs, Non-Paternity Events), as it may be difficult to distinguish a match that is an SDS (e.g. adoption, illegitimacy) from one that is due to Convergence.

Below is a list of these criteria and we will consider each one in turn. Some of these Markers of Potential Relatedness (MPRs) have nothing to do with DNA. If two people have the same surname, or the same unusual surname variant, or have a similar ancestral homeland, or even an ancestor with the exact same name, then these can be indicators that the two people are related. And because they don't rely on genetics I simply call them "traditional markers" as opposed to "genetic markers". 

MPRs for deciding if two or more people are related within the last 1000 years

In practice, the most useful indicators (or at least the ones I most frequently use) are Markers 1, 2, 6 and 7. And if a new project member is grouped on the basis of these "main" markers, it usually becomes apparent that they meet many of the remaining criteria also.

1. The members have the same surname

This is an obvious criterion, especially for surname projects that deal with hereditary surnames. If two people share the same surname, the next question is: are they related? And it would seem a reasonable supposition that there is a much higher probability that they are related on their direct male lines (within the last 1000 years) if they do share a surname than if they don't.

Problems tend to arise when there is some doubt over what is a valid surname variant and what is not. For example, are Malley and Malloy surname variants? Are Farrell and Farris surname variants? What happens when you get both types of variant testing positive for M222? Do you group them together or keep them apart? Only other MPRs (such as downstream SNP testing) can answer these questions.

2. The Genetic Distance (GD) between two people indicates a (very) close relationship

The threshold for "declaring a match" between two people varies with the number of STR markers tested (see below). These thresholds are arbitrary, but the intention is to get the right balance between false positives and false negatives - in other words, letting the wrong people in and keeping the right people out (known more technically as specificity and sensitivity).

Most people do the Y-DNA-37 test initially and I would usually feel very confident grouping together people with the same surname if their GD was 2/37 or less; and reasonably confident of grouping them together if the GD was 4/37 or less. Except in the instance where there is evidence of Convergence, as indicated (for example) by the terminal SNPs of their matches sitting on a wide variety of distantly related "upstream" branches of the Y-Haplotree (Tree of Mankind). We'll talk about this some more in item 7 below.

In addition, Convergence is a common occurrence in certain subclades, such as M222 and L226. When I see these terminal SNPs in a new project member, alarm bells start ringing, my level of conservatism increases, and I start looking to other MPRs other than Genetic Distance to decide if two people belong in the same genetic family.


This technique for grouping people together will miss outliers - people who do indeed belong in the same genetic family but whose ancestors branched away from the main group many many generations ago. For example, in the Gleeson DNA Project, several of the members of Lineage II (all confirmed to be related by Big-Y SNP testing) have a GD of 10/37 compared to other group members, and that would usually preclude them being grouped together.

3. The TiP24 score is >80% compared to the group modal haplotype

I don't use this marker so much anymore but it can be a useful way of assessing if a newcomer belongs in a given genetic family, especially if there is insufficient data regarding SNP markers among their STR matches. The potential benefit of this method is that it takes into account the varying mutation rates of STR markers whereas GD does not.

It involves generating a TiP Report between a new project member and the member closest to the modal haplotype for a given genetic family within the project, and then looking at the percentage probability of being related within 24 generations. We call this the TiP24 Score (for lack of a better term). If this is >80% (an arbitrary figure, which can be adjusted to suit your personal preference), then the newcomer can be considered to be "likely to be related" and therefore placed in that specific genetic family.

It is important to note that the use of the TiP24 Score is not an attempt to date when two people are related, merely to ascertain if two people are likely to be related. The TiP24 Score is simply an attempt to standardise GD comparisons, given that we know that a GD of (say) 4/37 on slow-mutating markers is much more significant than a GD of 4/37 on fast-mutating markers. The former (probably) indicates a much more distant relationship than the latter.

This techniques works best for those related within the last several hundred years, but will miss outliers. I have several people in the Gleeson DNA Project (confirmed to be related via SNP testing) whose TiP24 Score with other members is as low as 1%.

Also, the TiP24 Score is likely to be tripped up by Convergence (in the same way that GD is) and is therefore of limited utility in such circumstances.

4. There is a clear Genetic Distance Demarcation between project members within a genetic cluster & project members outside it

Administrators have access to a tool called the "Y-DNA Genetic Distance" tool. This permits comparisons between any person in the project and every other person in the project. Often times, there will be a clear demarcation between a newcomer's range of GDs to a particular genetic family and all other genetic families within the project.

In the example below, the newcomer matches 9 members of R1b-Genetic Family 2 with a GD ranging from 4/67 to 9/67. Thereafter, the GD jumps to 16/37 and higher. This stark demarcation in GD suggests strongly that the newcomer falls within R1b-Genetic Family 2.

This also suggests that Convergence is unlikely to be an issue here (otherwise we might expect to see a more gradual increase in GD values, rather than the jump from 9 to 16 that we see here).

This technique works best with 111 or 67 marker comparisons. Demarcations are much less obvious using 37 marker comparisons.

The GD between the newcomer & other members
shows a clear demarcation between
one particular genetic family and all others

5. Presence of Rare Marker Values or a Unique STR Pattern among genetic group members

The idea here is that if one or more people share a Rare Marker Value, then it stands to reason that they are more likely to be related to each other, especially if they all share the same surname.

Leo Little's spreadsheet of STR marker value frequencies is very useful for identifying those values which are particularly rare, even though the spreadsheet only covers six of the main haplogroups (E3a, E3b, G, I, J2, R1a, R1b). What constitutes "rare" is a moveable feast but a frequency less than 5% would not be unreasonable.

Usually these rare marker values emerge after several people have been grouped together. Any newcomers thereafter who share this rare marker value can be further assessed for membership of the specific genetic family wherein the rare marker value occurs. A famous example is Group B of the Wheaton Surname Project where 3 "rare" marker values occur within the first 12 markers (with incidences of 5%, 1% & 8% in the "general" R1b population). The chances of these occurring within the general population are 1 in 62,000. And therefore, any Wheaton who matches these 3 STR marker values can be automatically allocated to Group B (with 99.99% confidence). And they only need a 12-marker test to do so.

Leo Little's spreadsheet of marker value frequencies

An allied concept is that of the Unique STR Pattern (USP, also know by various other terms such as STR Motif). In short, these are a selection of STR marker values (usually between 3 to 8 in number) that are "unique" to just a few people within a surname project and which indicate that the people concerned are likely to be related to each other.

A good example from the Gleeson DNA Project shows that several members had relatively unique STR Pattern (USP) which predicted that they were related (Branch E and F below). This was later confirmed by SNP testing of the two branches.

Unique STR Patterns (USP) predict the existence of a Branch E and F (last 6 entries)
Branch E signature ... 464b=17, 607=14, 576=17
Branch F signature ... 391=10, 458=17, 459=9-9, 576=17

Robert Casey has developed this concept extensively and you can hear him talk about it in this video here.

6. SNP testing is consistent among the members of the particular group

The advent of Next Generation Sequencing (producing tests like the Big Y and the array of SNP Packs) has created a SNP tsunami. And as more people SNP test, their predicted red SNP is being converted to a green confirmed SNP on the project's Y-DNA Results page.

As a result, many groups within a surname project are having their "Terminal SNP" characterised. And this allows us to compare any SNP markers that the newcomer has tested with the SNP markers that characterise the various groups within our surname project. If they are discordant, then the newcomer is ruled out from membership of those particular genetic families. But if they agree with each other, especially if they are SNPs quite far downstream, then this is further supportive evidence that the newcomer belongs in a specific genetic family.

The phrase terminal SNP is a bit of a misnomer. It should be restated as "current terminal SNP" and simply means the "most downstream" SNP marker that you have currently tested. And what is meant by "most downstream"? Imagine the Tree of Mankind (the Y-Haplotree) as starting with genetic Adam (upstream) about 250,000 years ago and the various branches emerging from him and continuously branching over many thousands of years into finer and finer "more downstream" branches, until these finer branches start approaching the origin of surnames (roughly 1000 years ago) and a genealogical timeframe. So your "most downstream" branch would be the branch characterised by your "most downstream" SNP marker ... which in turn is determined by your current level of SNP testing. For example, your Y-DNA 37 STR results will predict which Haplogroup branch you sit on (let's say it is R-M269, which arose about 13.5K years ago), and the R-M269 SNP Pack will take you a little further down Branch R (say to Z255, 4000 years ago), and the R-Z255 SNP Pack, will take you even further downstream (maybe to 2000 years ago), but the Big Y test will take you the furthest (maybe down to 500 years ago).

In the example below, all the green confirmed SNPs sit below the SNP marker that defines Gleeson Lineage II, namely A5631. Therefore any newcomer who matches any of these SNPs (even if he has a large GD to everyone in the project) can be reliably grouped into Lineage II. The abbreviated SNP Progressions (or SNP Signatures) for each of the individual SNPs is detailed below:

  • R-Z255 > Z16437 > Z16438 > BY2852 > A5631 > A5629
  • R-Z255 > Z16437 > Z16438 > BY2852 > A5631 > A5629 > BY5706 
  • R-Z255 > Z16437 > Z16438 > BY2852 > A5631 > A5629 > BY5706 > BY5707 
  • R-Z255 > Z16437 > Z16438 > BY2852 > A5631 > A5629 > A5628 > Y16880

Predicted SNPs (red) and Confirmed SNPs (green)

The predicted red SNPs are almost always much further upstream on the Tree of Mankind than the green confirmed SNPs. Think of the upstream SNPs as closer to Genetic Adam (250,000 years ago) and the downstream SNPs as closer to a genealogical timeframe (say, 1000 years ago).

7. SNP predictions are consistent (Matches’ Terminal SNP Analysis) 

NB: SNP Predictions does not mean the red predicted SNP you get in the Haplogroup column (see figure above) when you first get your Y-DNA-37 results. It refers to SNPs much further downstream than that, usually within the last 5000 years and frequently within the last 2000 years.

If a newcomer to the surname project has not undertaken downstream SNP testing, it is still possible to guess what his downstream "terminal SNP" will be by simply analysing the terminal SNPs of his STR matches. I call this the Matches' Terminal SNP Analysis. It is a relatively simple technique that takes a little time to complete. Here are the steps in the process:

1) First, open up the Y-DNA Matches page and adjust the Matches Per Page setting so that all the matches are on the one page.

2) Next click on the heading in the Y-DNA Haplogroup column so that all of the matches are sorted by their terminal SNP.

3) Make a list of all the terminal SNPs (you can ignore the SNPs that are way upstream e.g. M269, P312, L21, etc)

4) Find out where each SNP sits on the Y-Haplotree, and (most importantly) the major subclade to which it belongs. You can do this by either of two ways: a) launch FTDNA's Haplotree, press Ctrl+F (Cmd+F on a Mac) and enter the SNP name. Once you find it, trace the branch back up to the previous branching point, make a note of the SNP there, and repeat the process until you arrive at a known subclade SNP; or b) google the following: "ytree" and the SNP name ... and this will bring you to the relevant page on the Big Tree. Then simply copy and paste the SNP Progression from the top of the page.

A google search for: ytree a5631

5) Both of the above methods will result in you having a SNP Progression for each SNP in the Matches List (see example below). If all (or most) of these SNP Progressions fall below a certain sublcade, then the likelihood is that the newcomer will also test positive for some SNP below this subclade level. It may even be possible to predict that he sits on one of maybe two or three "way downstream" branches. And this can be strong supportive evidence that he is related to certain project members and should be grouped in a particular genetic family.

If on the other hand, the various SNP Progressions associated with this list of SNPs indicate that the newcomer is matching to multiple distinct upstream branches of the Haplotree, then no firm conclusions can be drawn about the newcomer's likely terminal SNP and therefore this information cannot be used to help place him in a specific genetic family.

6) As a result of this analysis, I may write to the newcomer and suggest they skip the upstream SNP Pack (e.g. R-M269) and move down to the more relevant downstream subclade SNP Pack (e.g. R-Z255) and purchase that one ... warning them that there is a 1% chance that my assessment may be wrong (but I haven't been wrong yet).

Output of the MTSA for a new project member
(he was advised to do the R-L1065 SNP Pack)

There are SNP Packs available for most of the major subclades and it is important to know what these are. You can see a list of them by logging in to your FTDNA account, clicking on Upgrade, then Advanced Tests, then SNP Packs from the drop-down menu.

Surprisingly, this analysis works best at the 25-marker level (because there are usually too few matches at the 37, 67 and 111 marker levels).

Occasionally I will have to use to check for the existence of equivalent SNPs or alternative names (if the SNP in question does not turn up in the FTDNA Haplotree or the Big Tree).

8. The same surname variant is predominant in a genetic group

This usually emerges after the new project member has been grouped on the basis of the previous MPRs described above. This serves to support and validate the decision to group the newcomer in the specific genetic family.

9. The same MDKA location is present in the particular genetic group

As above. This serves to illustrate how essential it is to encourage all project members to include the birth location of their Most Distant Known Ancestor (MDKA / EKA) in the Genealogy section of their personal FTDNA webpages. After their surname, their ancestor's birth location is the single most important piece of information.

Always include the birth location of the EKA / MDKA

10. The same MDKA is present in the particular genetic group

This is the ultimate validation that the grouping based on the preceding MPRs is valid and accurate.

For a more detailed discussion of these various criteria, watch the video below. This is suitable for beginners, those who have already done the Y-DNA test and want to find out what it means, and for Surname Project Administrators.

Maurice Gleeson
June 2017

Tuesday 13 June 2017

WDYTYA 2017 - videos going online

This year was the last year of Who Do You Think You Are? - Live! The event was an annual staple of the British genealogical calendar for the last 10 years. Starting in 2008 in Olympia in London, it moved to the National Exhibition Centre in Birmingham in 2014. The event attracted thousands of attendees year on year, and in 2009 Brian Swann, ISOGG-UK representative, persuaded FamilyTreeDNA to sponsor a stand at the event.

Shortly thereafter, the DNA Workshop began. And for the past several years this has been kindly sponsored by FamilyTreeDNA and run by volunteers from ISOGG. Each year the lecture schedule has attracted a host of international and local speakers, both academics and citizen scientists. And in addition, videos of the presentations have been made available free of charge on our dedicated YouTube channel as a service to the genetic genealogy community.

It is sad to see the demise of the WDYTYA event. It was a wonderful way of keeping in touch with friends and colleagues, and everyone looked forward to the manic three days of early mornings and late nights. Hopefully another annual event will rise to take its place. Nevertheless, at least the videos of the presentations will serve as a lasting legacy of the ten year run of WDYTYA.

The last batch of videos ever are ready to be uploaded to the YouTube channel and this will happen every Monday and Thursday over the coming 6 weeks or so. Three videos have already been uploaded and are attracting a large audience:

The Science of Admixture Percentages (Garrett Hellenthal)

DNA, emigration and shipping (Brian Swann)

Autosomal DNA demystified (Debbie Kennett)

The schedule of lectures from this year's event are indicated below (click the image to enlarge). Most of the presenters gave permission to upload their lectures and big thank you is due to the speakers for their generosity.

And this year we managed to get better audio recordings than ever before. Who would have thought that dangling a small microphone in front of the loudspeaker and recording a separate audio track on your iPhone would be the best way of conquering the ever-present background noise from the 10,000 people in the auditorium?!


Maurice Gleeson
June 2017

Thursday 1 June 2017

Convergence - quantifying Back & Parallel Mutations (Part 1)

In a recent post I explored the concept of Convergence and made the point that the mechanism by which Convergence arises is via a combination of Parallel Mutations and Back Mutations in the STR marker values. These mutations are changes that occurred at some time in the past but because they remain hidden to us in the present, we cannot tell when they occurred or how frequently they occurred just by looking at two sets of STR results from people living today.

However, there is a way around this problem. Or at least a partial solution.

By using a combination of STR data and SNP data we can build a Mutation History Tree that is a more accurate representation of the branching structure of the "family tree" for a specific genetic group. And this type of tree allows us to more easily (and more accurately) spot Back Mutations and Parallel Mutations.

I did this for one particular genetic family in one of my surname projects - the North Tipperary Gleeson's (Lineage II of the Gleason DNA Project). This tree is a "best fit" tree, by which I mean a tree constructed in such a way as to explain the STR & SNP data in the most parsimonious way i.e. with the fewest number of branches that will accommodate or "fit" the data. This approach is also called the "maximum parsimony" approach and is often used when building cladograms or phylogenetic trees. The Mutation History Tree (MHT) is simply another type of cladogram. You can read about the process of how the tree was developed in this blog post here and subsequent posts.

But a key point here is that this "best fit" tree is likely to change as more data becomes available. And to illustrate this point, I'm going to compare the current version of the tree (Dec 2016) with the next version that is being prepared following the recent availability of new data from 12 sets of Z255 SNP Pack results.

Below is the current version of the MHT for Lineage II. By comparing each mutation in the tree with every other one, we can identify which mutations are Back Mutations (occurring on a single line of descent) and which are Parallel Mutations (occurring on two or more lines of descent). I have highlighted the Back Mutations in yellow and the Parallel Mutations in green.

Back Mutations in yellow, Parallel Mutations in green
from Gleeson Lineage II MHT (version Dec 2016)

Parallel Mutations occur in the following lines of descent:
  • CDYb 40-39 ... A, E, D, F (4 times)
  • CDYa 39-38 ... A, B, C, F (4 times)
  • 464c 17-16 ... A x2, D (3 times)
  • 461 12-11 ... A, B (2 times)
  • 576 18-19 ... A, D (2 times)
  • 390 23-24 ... A, B, C (3 times)
  • 390 24-23 ... B, C (2 times)
  • 456 16-15 ... B, D (2 times)
  • and so on ...
Back Mutations are more difficult to count, and to conceptualise. Whether you consider the value as mutating forward or back is entirely dependant on your reference point. If our anchor is the upstream Z255 branch, then the original value of marker 390 (for example) is 24, mutating (forward) to 23 on the Z16438 branch, and then back to 24 (in parallel) on Branches A, B & C, and then back to 23 (again in parallel) on Branches B & C. So there are several points to make here:
  • this is in fact a Back Mutation that occurs in parallel in 3 separate lines of descent. It is thus both a Back Mutation (relative to its earlier value of 24 on the Z255 branch) and a Parallel Mutation, occurring at (presumably) different time points in Branches A, B & C. It is thus coloured yellow and green.
  • It can also be considered a Triple Mutation relative to the Z255 branch - in the sense that it mutates forward to 23 then back to 24, then back to 23 again. But what happens if it flips forward and back 5 times? What would we call that? And what do we call it if it goes two steps forward and one step back? This is where terminology fails us. I'm not sure if there is a standardised way of describing these different kinds of mutation (if there is, please leave a comment below).
  • the mutation 390 24-23 occurs in Branches B & C ... relative to its value of 24 in the Z255 branch, this could be considered a Parallel Forward Back Forward Mutation ... for Pete's Sake!!

But if we just focus on the Back Mutations that occur downstream of the branch characterised by the STR mutation (710 36-37), just above the A5627 SNP Block. This "710 branch" incorporates all the Gleeson's of Lineage II, from Branch A to F.* On this overarching branch for Lineage II, the value of the STR marker 390 is 23 and Back Mutations are as follows:
  • 390 24-23 ... B, C ... this is the only Back Mutation below the "710 branch"
  • And it is also a Parallel Mutation
  • All the other yellow Back Mutations are relative to the upstream Z255 branch, and not our downstream "710 branch", and so are not counted in this particular exercise.

So, let's generate some statistics from these numbers:
  • The total number of mutations below the "710 branch" (irrespective of whether they are forward or back) is 71.
  • There are 69 Forward Mutations (i.e. away from the original value of the relevant marker on the "710 branch")
    • 31 Forward Mutations show an increase in the number (e.g. 9 to 10)
    • 38 Forward Mutations show a decrease in the number (e.g. 9 to 8)
  • There are 2 Back Mutations 
    • both Back Mutations show a decrease in the number (i.e. 24 to 23)
  • There are 26 Parallel Mutations
  • Forward Mutations outnumber Back Mutations by a ratio of 35.5 : 1
  • Parallel Mutations outnumber Back Mutations by a ratio of 13 : 1
  • There are 16 people in this tree, and if we make the big assumption that the "710 branch" starts 1000 years ago (i.e. roughly at the time of the introduction of the Gleeson surname), then over the course of 1000 years, the rate of each type of mutation is (crudely) as follows:
    • Forward Mutations = 69/16 = 4.3125 mutations per "line of descent" per 1000 years
    • Back Mutations = 2/16 = 0.125 mutations per "line of descent" per 1000 years
    • Parallel Mutations = 26/16 = 1.625 mutations per "line of descent" per 1000 years

These are crude estimates but they give some idea of the relative importance of Parallel Mutations compared to Back Mutations. And applying this information to the phenomenon of Convergence, it would seem that Back Mutations play a very minor role compared to Parallel Mutations.

This conjecture is supported by some recent modelling work undertaken by Dave Vance and written up for the L21 Yahoo Discussion Forum. In Dave's simple model, which is an extremely useful basis for further discussion, the "average tree" could expect to have a ratio of Parallel to Back Mutations in the range of 25:1 to 50:1.

This is a lot higher than what I have shown in my MHT for the Lineage II Gleeson's, but this can be partly explained by the fact that there are only 16 people in my Gleeson sample, and we are looking at (perhaps) only the last 1000 years. I would predict that the ratio will increase further as 1) I add more people to the sample; and 2) the duration of observation is extended backward from 1000 years ago (the 710 Branch) to 4300 years ago (the Z255 Branch).

In subsequent posts we will see how these calculations stand up when we add in additional data from 12 SNP Pack results and reconfigure the MHT for Gleeson Lineage II into the next version of the "best fit" model. And we will also attempt to quantify the total number of Back & Parallel Mutations below the upstream marker Z255. And lastly, we will attempt to quantify Convergence itself.

Maurice Gleeson
June 2017

* the Big Y results of a 10th member of the group indicate that this branch is characterised by the SNP A5631 although this result is not reflected in this version of the MHT