23andMe data file changes

A few recent submissions to the Project have alerted me to the fact that 23andMe has been making changes to its data download. It is unfortunate that such changes are made apparently "silently", as they may negatively impact third-party tools built around the 23andMe data.

Apparently, the orientation of some SNPs has been changed. This should not be a problem for DIYDodecad, as it handles orientation of different companies automatically. A different problem is that apparently some SNPs have been dropped from the data file altogether. This is a problem if you are not using the latest 2.1 version of the DIYDodecad software, so you should upgrade to 2.1. It seems that about 80 of the SNPs expected by the 'dv3' calculator have been removed from the data file download, and these will appear as "absent". I do not expect 80 absent SNPs to have a huge impact on results, as they make up less than 0.1% of all SNPs in dv3.

The change in format has other consequences as well; as many of you know, I have been working on Dodecad v4 for some time now. This would use common markers between 23andMe v2 and v3 platforms and Family Finder Illumina platform. However, I will now have to backtrack on it, to make sure that the marker set used is actually consistent with people's current 23andMe downloads.

If you have a fresh 23andMe downloaded file and DIYDodecad 2.1 and you are unable to run 'dv3' or any other Project calculators, drop me a line.

A note on the eligibility criteria of the Dodecad Project

1. Why I do not accept relatives

Relatives are much more similar to each other than they are to other people. When an ADMIXTURE run includes a pair of close relatives, then there is a high risk of the two relatives forming their own component to which they belong near 100%. Thus, (a) they learn nothing about themselves, other for the fact that they are related, and (b) other people have some membership in that particular component which does not have any real anthropological interest.

The same is true when one performs dimensionality reduction like multidimensional scaling (MDS) or Principal Components Analysis (PCA). It is highly likely that a pair of relatives will be distinguished from everyone else along a dimension. That dimension is "wasted", and it provides no real anthropological information other for the fact that a pair of people are related to each other. If this is combined with a clustering algorithm such as the Galore approach, then the pair of relatives will form their own anthropologically meaningless cluster.

The Dodecad Project is an anthropology project; it is not a genealogy project. There are other projects and experts out there who can help you interpret your data from a genealogy perspective.

2. Why I accept people with 4 grandparents from the same European, Asian, or North African ethnic group or country

I do so for two reasons, a scientific one, and a practical one:

First, such individuals allow me to gradually build a database of genetic variation for my region of interest, namely Eurasia. As of this writing, there are 387 participants in the project, out of which I have managed to form 19 populations, each of them consisting of at least 5 individuals from a particular ethnic/linguistic/national/geographical group such as Greek/Arab/British/Scandinavian, for a total of 209 individuals.

There are many participants who belong to a particular group for which there are not 5 participants yet; and, there are also many who have a mixed background and whom I have included in the Project either during some time-limited submission opportunities, or because they asked me to, making a good case for why they should be included.

The second reason is practical, as the customers of 23andMe are probably largely American, and many non-1st generation Americans are descended from multiple ethnic groups. Even though a large part of the analysis process is automated, I still need to spend some time on every submitted sample to download it, record it, convert it, to assign some processing time on my computing systems for the analysis, and then spend some more time to create the results barplots and materials.

I simply don't have the time to do so for the large pool of 23andMe customers, but I can, occasionally, accept individuals of mixed heritage during short submission opportunities.

What to do if you are not eligible

If you do not meet the eligibility criteria, and think that I should include you in the Project, the thing to do is to write to me, laying out the reasons why. Here are some scenarios that I might very well consider:

1. Adoptees with no knowledge of their origins.
2. People of mixed heritage who belong partly to an unsampled group. Let's say you are 50% English+50% Parsee, then I might consider this sample as I have no data on Parsees.
3. People of mixed origins who have data for both their unadmixed parents (and thus provide two samples to the project), or a series of their unrelated relatives, I may very well analyze the admixed individuals as well.
4. People of mixed heritage from Greece and its environs (i.e., Italy, the Balkans, and Anatolia), e.g., 50% Gypsy+50% Bulgarian, or 50% Croat+50% Serb or 75% Turk+25% Albanian would all probably be accepted if they contacted me.

Open-ended submission opportunity for 23andMe data (#2)

Who is eligible

Everyone who is of European, Asian, or North African ancestry and all four of his/her grandparents are from the same European, Asian, or North African ethnic group or the same European, Asian, or North African country.

Please do not submit samples of relatives, as these make analysis difficult. I consider 2nd cousins and closer relatives to be related.

I am sorry that I can't process everyone's data, so if you don't fit the above criteria and you feel you should be included, feel free to write to me (but don't send me your data!) and I will keep it in mind. Also, you can subscribe to the feed for future opportunities to submit your data.

What you will receive

You will receive the standard K=10 analysis results such as this, and you will also be eligible for other types of tests such as Galore analysis or regional studies

Your raw data or genealogical information will not be shared or distributed in any manner, and it will not be analyzed for any other purpose than assessment of ancestry (i.e., not for any physical or health-related traits). It will be identified by a unique ID, known to you and me, and results will be posted in the blog using that ID. I will continue to analyze your data for ancestry, and new results will be posted using that same ID. Also, I will report aggregate results for populations with at least 5 participants.

What to send

Send your zipped autosomal data to dodecad@gmail.com. Also let me know something about your ancestry (e.g., ethnic group, country of origin of grandparents, or anything else that might be useful).

23andMe v3 data

First, note that submission to the project is closed. Any data I receive will go to the back of the line for processing; moreover, I give no guarantees that I will process any unsolicited data sent to me, as I generally process data from selected groups and not everyone.

Second, I had the opportunity to look at the v3 data as reported by 23andMe. By my count there are 936,845 markers in the v3 platform, including 168,554 that are in common with the ~177k subset I normally use for 23andMe v2 data. Thus, while most of the v2 SNPs are part of the v3 platform, some of them are not. I anticipate no problems in integrating v3 data with the existing database of the Project.

Open-ended submission opportunity for 23andMe data

In the spirit of the holidays, I am open to receiving 23andMe data for analysis. This will be an open-ended call rather than one with a deadline, and I will announce when it will be over in this very post.

Who is eligible

Everyone who is of European, Asian, or North African ancestry and all four of his/her grandparents are from the same European, Asian, or North African ethnic group or the same European, Asian, or North African country.

Please do not submit samples of relatives, as these make analysis difficult. I consider 2nd cousins and closer relatives to be related.

I am sorry that I can't process everyone's data, so if you don't fit the above criteria and you feel you should be included, feel free to write to me (but don't send me your data!) and I will keep it in mind. Also, you can subscribe to the feed for future opportunities to submit your data.

What you will receive

You will receive the standard K=10 analysis results such as this, and you will also be eligible for other types of tests such as Galore analysis or regional studies

Your raw data or genealogical information will not be shared or distributed in any manner, and it will not be analyzed for any other purpose than assessment of ancestry (i.e., not for any physical or health-related traits). It will be identified by a unique ID, known to you and me, and results will be posted in the blog using that ID. I will continue to analyze your data for ancestry, and new results will be posted using that same ID. Also, I will report aggregate results for populations with at least 5 participants.

What to send

Send your zipped autosomal data to dodecad@gmail.com. Also let me know something about your ancestry (e.g., ethnic group, country of origin of grandparents, or anything else that might be useful).

Outliers in the Dodecad Project (23andMe data)

As promised, I have started to investigate outliers among Dodecad Project members. I used NNclean as implemented in the prabclus package to find data points that had a great distance to their nearest neighbor among either Dodecad Project members or the standard 692-individual panel I use in the Galore analysis.

To make a long story short, here are the IDs identified as outliers:

"DOD157" "DOD168" "DOD169" "DOD036" "DOD048" "DOD088" "DOD034" "DOD030" "DOD060" "DOD132" "DOD128" "DOD175"

An outlier is someone who is not very close to any other individual and hence does not really "cluster" with anyone. Thus, it is recommended to remove outliers prior to clustering, as otherwise they will form makeshift clusters that don't really have a good meaning.

Looking at the individual spreadsheet reveals that many of these outliers have very unusual ancestry. This falls under two categories:

1. Recent admixture between geographically separated populations
2. Being the only member from an unsampled population

In the first case, admixed individuals fall in the "empty space" between their parental clusters, and thus do not cluster with anyone else, unless a person with a similar type of admixture happens to also be in the dataset.

In the second case, there are no members of the individual's group. Sometimes, if a group is close enough to another, this is not a problem, but there are many distinctive population groups for which that is not the case.

While outliers will be removed from some analyses, their outlier status will continue to be evaluated as new reference populations, or Dodecad Project members are added.

Galore analysis improved, plus K=56 results of Clusters Galore analysis for Dodecad Project members (up to DOD236)

For background, please read the post on the K=48 analysis and links therein.

As I mentioned in the previous posts, my technique depends on the use of MDS to reduce the dimensionality of genomic data from 177,000 SNPs or so to a few dozen dimensions capturing most of the variance.

Subsequently, mclust a state of the art clustering algorithm is applied on the MDS representation: this iterates between choices of K, the number of clusters, trying clusters of different shape, volume, and orientation, and chooses the optimal clustering, maximizing the Bayes Information Criterion. In simpler terms, it finds as much detail as possible in the data but penalizes too ornate models and avoids finding "ghost" clusters that are not really supported.

These are clusters derived from data of unlabeled individuals. The only human input into the process is the number of MDS dimensions to retain.

In my previous K=48 analysis, I retained 30 dimensions, but I also noted that this is not really optimal. Choosing more, or less, dimensions might lead to even better resolution (higher K).

More dimensions = more possible ways to distinguish between individuals, but also, possibly, more noise, as individuals might not be "clustered" in them.

Fewer dimensions = less possible ways to distinguish between individuals, but also, possibly, less noise from the uninformative higher dimensions.

Thus, the question arises: how many dimensions to retain?

Here is a plot of the optimal number of clusters inferred, depending on how many dimensions I chose to retain:

As you can see, when a few dimensions are retained, relatively few clusters are inferred, while as the number of dimensions goes beyond a certain point, the number of clusters starts to decrease again, as more noise is added (*)

The number of clusters peaks (see figure) at 16 and 22 dimensions retained; both of these produce 56 different clusters in the optimal solution.

Here are the results for Dodecad Project members (up to DOD236) with K=56 and 16 dimensions retained. In comparison to the previous K=48 analysis, we are now able to:

1. Split CEU White Utahns (#1) from French (#15)
2. Split CEU White Utahns (#1) from continental Germanics (#14)
3. Split French (#15) from Spaniards (#2)
4. Split Armenians (#7) from Turks (#19)
5. Split Slavs (#23) from Balts (#26)
6. Split Cypriots (#30) from Sephardic Jews (#21)

The most astonishing finding is, however, at least for me, the emergence of a cluster (#16) comprised in the great majority by people from Greece and Southern Italy, with very few individuals from elsewhere. Notice that #16 has absolutely no representation in the reference populations, which lack South Italians and Greeks.

Once again, I urge participants to help themselves and others by leaving a comment in the ancestry thread.

(*) The change is not, however, smooth. The more general problem is to choose which dimensions to retain, rather than choosing how many of the first ones to retain. The first few dimensions of MDS capture a decreasing portion of variance, but the data are not guaranteed to be "split" in them. However, this is a much harder problem, as we have to figure out (i) how many dimensions to retain, and (ii) which ones. Even if we fix (i), by choosing to retain, e.g., 10 dimensions, we still have to choose which 10: this is close to half a billion different combinations of which 10 to choose from the total of 38 possible candidate ones.

Results of Clusters Galore analysis for Dodecad Project members (up to DOD236), K=48

This is the result of the new type of ancestry analysis I have recently devised. For background, please read:

• Clusters galore: extremely fine-scale ancestry inference
• Clusters galore with Dodecad populations
• Clusters galore: less is more, or, pushing the limits of ancestry inference

In total 894 individuals were included in this analysis, 202 Dodecad Project members and 692 from the published references. 30 MDS dimensions were retained, and mclust was run with a maximum number of clusters = 60. In the optimal solution, 48 clusters were inferred.

At the beginning of the results spreadsheet are the 202 Dodecad Project members and their probabilities of belonging to the 48 clusters. This is followed by the 36 reference populations, listing how many individuals from each one were assigned to each of the 48 clusters.

To help you interpret these results, you might want to consult the individual and population ADMIXTURE spreadsheets, as well as the information Project members have chosen to reveal about themselves. Feel free to add your own information in that thread.

Here are the results for some people who have chosen to reveal information about themselves:

• Spencer Wells is DOD162 and he has 28% probability of being in the CEU (White Utahn)/French cluster #1, and 72% probability of being in cluster #3 which is CEU (White Utahn) specific (in the reference populations) and in which 29 Project members are also assigned (almost all of them Northwestern Europeans)
• Razib of Gene Expression is DOD075 and is assigned in cluster #15, which includes Gujarati and North Kannada individuals
• pconroy, submitted three samples, DOD097 is Sicilian fall in cluster #9 to which many Cypriots and Sephardic Jews belong, and many Project members of South Italian/Sicilian background; DOD098 and DOD099 are his Irish parents: DOD098 is almost evenly split between #1 and #3 (like Wells), and DOD099 is 97% in #3
• Adriano Squecco DOD139 is North Italian and is in cluster #2, in which 25/25 reference Tuscans and 10/12 reference North Italians belong
• Lacko DOD083 is 100% southern Polish and he falls in cluster #20, in which all reference Belorussians and Lithuanians fall
• Mike Maddi DOD021 is Sicilian and is in cluster #9 like DOD097, showing some probability (13%) of also being in #2 like Adriano Squecco
• An Anonymous Pole falls in #20 like Lacko and the reference Slavs
• Ilmari DOD003 and Ari DOD131 are Finns and fall in cluster #13. This is an interesting one, as it does not occur at all in the reference populations; I'll let you guess what population it's centered on.
• Eastara's mother (DOD025) is Bulgarian and falls in cluster #10, which is centered in Romanians in the reference populations, but note that there are also 13 Dodecad Project members who fall in it, many of them from different parts of the Balkans. I hope more people from the Balkans will contact me for inclusion in the project, as I am sure that finer-scale can be achieved there with increased participation.
• Basar (DOD049) is half-Anatolian Turk and half-Laz. He falls in the cluster encompassing Armenians and Turks in the reference populations.
• Bubba (DOD066) is North German with a pinch of Danish and falls in cluster #3
• afpjr (DOD014) is half Greek and half Italian/Sicilian; he falls in cluster #9 (95%)
• Francesc (DOD217) is Catalan and falls in cluster #17, centered on Spaniards in the reference populations.

The Project Greeks and Mixed Greeks fall in clusters #2, 9, 10 tying them to Italy and the Balkans, as might be expected. I hope more Greeks will decide to participate in the Project, so we can discover more interesting patterns in our population.

I cannot stress enough how revealing non-identifying ancestry information in the relevant thread will help both yourselves and others make better sense of these and future results.

There are clusters composed entirely of Dodecad Project members (e.g., the aforementioned one), and others which are centered on one or two reference populations, but encompass a wider variety of non-represented populations. So, please take the time to leave a comment in the ancestry thread.

This is not the end of the story. There are more clusters to be discovered in the data; the inclusion of 200+ new samples in this analysis has caused new clusters to appear and distinctions that were previously detected to "fold back" (e.g., between Armenians and Turks). I am currently investigating how the choice of number of MDS dimensions to retain affects the number of inferred clusters in the optimal solution.

Clusters galore with Dodecad populations

Number of individuals assigned to each cluster can be found in the spreadsheet. Populations in italics are composed entirely from Dodecad Project members.

Please read the post in Dienekes' Anthropology Blog to see what this type of analysis means.

47 MDS dimensions were retained, and the optimal number of clusters was 35. Retaining less or more dimensions may alter this number, as after a certain point extra dimensions only contribute noise to the analysis; this is a matter of investigation.

It is hardly practical to comment on all 35 clusters, so I will limit myself to a few observations:

• Turks, Armenians, and Assyrians fall in cluster #1
• Scandinavians, White Utahns, Germans, and some French fall in cluster #2
• Portuguese, French, North Italians, Tuscans, Spaniards, and Romanians fall in cluster #3
• Greeks, South Italians/Sicilians, Cypriots, and Sephardic Jews from Bulgaria and Turkey fall in cluster #4 (but see note)
• Finns fall in cluster #5
• Almost all Ashkenazi Jews fall in #6
• All Dodecad Project Russians, plus reference Lithuanians and Belorussians fall in #9

8 Greeks fall in cluster #4 and 2 in cluster #3. However, many of the ones who fall in #4 also have some non-trivial probability of falling in #3. Probabilities for all other clusters are less than 0.1%. All Project Greeks can write to me to learn their exact probabilities.

Of course, it should be noted that:

1. If two populations can be perfectly distinguished from each other, then there are genetic differences between them (they split from each other some time ago, they underwent different types of admixture, etc.) allowing the clustering algorithm to detect their differentiation
2. If two populations cannot be distinguished from each other, this does not mean that they are not indistinguishable in principle; it does mean, however that through either common ancestry or very similar patterns of admixture, they have become quite similar to each other in the Eurasian context.

If you are a Dodecad Project member (23andMe data) from one of the populations in italics and are wondering which cluster you fall in, first check whether all individuals from your population fall in the same cluster, in which case you already know the answer.

Otherwise, you may write to me, with your DOD number, and I'll tell you.

Results for Greg and Lilly Mendel and Genomes Unzipped individuals added

Admixture proportions can be found in the spreadsheet

Greg and Lilly Mendel (i.e., Linda Avey, 23andMe co-founder, and her husband) and all populations:



Individual bars for Genomes Unzipped individuals. I had previously studied these in a test that lacked "South Asian" and had slightly different categories for West Eurasians.

Introducing the Dodecad ancestry project

Welcome to the Dodecad ancestry project!

The Dodecad ancestry project, named for the Greek word for "group of twelve", aims to provide detailed ancestry analysis, primarily for Eurasian individuals. Please read on, if you want to participate; if not, please subscribe to the feed to keep up with the project's progress.

The project was started by Dienekes' Anthropology Blog.

1) Project goals

The Dodecad ancestry project has two goals:

1. To provide detailed ancestry analysis to individuals who have tested with 23andMe; other testing companies may be included in the future.
2. To build samples of individuals for regions of the world (e.g. Greeks, Finns, Albanians, Southern Italians, etc.) currently under-represented in publicly available datasets.

I neither endorse nor am I affiliated with any genetic testing company. I have chosen to base the project on 23andMe results, because (i) I perceive that quite a few people have used the service, (ii) the Illumina genotyping platform it uses has substantial overlap with the publicly available datasets on which my analysis depends.

2) Data privacy statement

Your raw genetic data will not be shared with anyone.

It will not be analyzed for anything other than ancestry or admixture. No analysis of physical or medical traits will be performed.

Individual-level results will be revealed with only a unique ID, without any further information about the identity or origin of each participant.

3) Who is eligible to participate

Due to my inability to process a large number of samples, at present, only the following groups are eligible to participate in the project's current pilot phase:

1. Greeks (not necessarily from Greece: Cypriots, Pontic Greeks from the former USSR, North Epirotes, Griko speakers from Italy, Muslim rumca speakers from Turkey, etc. are all accepted)
2. People from the Balkans
3. People from Anatolia
4. People from the Caucasus
5. Italians
6. Non-Indo-European speakers from Europe (e.g., Finns, Hungarians, Basques)
7. Scandinavians and Icelanders
8. Iranians
9. Armenians
10. Jews from Italy, the Balkans, or Anatolia
11. Assyrians
12. Arabs

Samples should be received by the end of October 2010. There may be a new opportunity to submit your data after that, which will be announced in this blog.

If you are uncertain whether your sample can be included in the project, please write to me at dodecad@gmail.com to inquire.

Close relatives should not submit all their samples. If you and your relatives have tested, please submit independent samples. For example, if you have data for you, your father, and your mother, it is ok to submit either (i) your own data, or (ii) your father and mother's data -- provided that it is not a consanguineous (e.g., cousin or uncle-niece) marriage.

4) What to send

Your compressed genotype file from 23andMe and as much information about your ancestry as you wish to reveal. It is necessary, however, that you tell me at least the country of origin of most of your ancestors and their ethnicity. Information about spoken language, religion, may also be useful.

5) Where to send it

Send it to dodecad@gmail.com. I will respond with a unique identifying code of the form DOD001, DOD002, and so on. The results will then be posted in the blog with that ID.

6) What you will receive

You will be included in an ADMIXTURE analysis together with other project participants and publically available populations, and the results will be published in this blog. Your sample will only be identified by your ID. If a group (e.g., Greeks) has at least 5 participants, I may also post the average admixture proportions for that group.

The following figure and table gives an idea of what to expect. Project members can expect to get a bar of their own, and a list of their ancestral proportions.

The number of components and population samples may vary over the course of the project.

7) Project updates

All project updates will be announced and presented in this blog. Additional commentary on the project may be posted in Dienekes' Anthropology Blog. I may also post occasionally on twitter about the project's progress.

8) Feedback

I encourage feedback about the project from participants or prospective participants. Please address it to dodecad@gmail.com