Australia

France

Ireland

Ivory Coast

Mali

Romania

Saint Lucia

Senegal

Solomon Islands

Sweden

United Kingdom

United States of America

Three months ago, realizing we had way too many duplicates in our database, we stopped importing lists to take a closer look at our algorithms. We now believe&nbsp;we have fixed the most glaring problems. After wiping out the entire database, we’ve started reimporting&nbsp;US records.What have we&nbsp;learned?First, we put&nbsp;a greater emphasis on dates. We used to&nbsp;accept&nbsp;long lists of names with no&nbsp;birth-related info, thinking they formed a great&nbsp;frame&nbsp;to later be fleshed out&nbsp;with details. The trouble is most names are very common. How do you know if “Roger Brown born in Atlanta” and&nbsp;“Roger Philip Brown” are the same person? Add the date of birth to the equation and the probability that they are is multiplied by 25,000. This has allowed&nbsp;for lots of records to be merged. Most lists available online do not include DOB,&nbsp;but you’d&nbsp;be surprised how many we’ve been&nbsp;able to find.Second, we no longer infer the year of birth. When processing the results of the French baccalauréat,&nbsp;for instance,&nbsp;we used to infer that most candidates were between 17 and 21 years old, or, in our parlance: 19 +/- 2. We’ve dropped this practice because a mere 5% of candidates outside of these bounds translated&nbsp;into tens if not hundreds of thousand&nbsp;errors in our base. The only exception we make is to deduce&nbsp;the birth year of a deceased&nbsp;person&nbsp;based on their age. If someone died in 2022 at the age of 72, we can say with absolute certainty&nbsp;that&nbsp;they were born in 1949 or 1950.Third, we stay away from unofficial lists. They simply present too many risks, from the use of nicknames (“Bill” for “William”) to truncated names (“Jose Michel Gonzal”) to their limited size&nbsp;(some are simply too short to make them&nbsp;worth our while). In a nutshell:<ul><li>We love electoral and vital records, obituaries, exam results, lists&nbsp;of social aid recipients and&nbsp;certified professionals.</li><li>We like professional elections colleges, lists of sports federation members and&nbsp;college graduates.</li><li>We are wary of marathon results, trade show and&nbsp;conference attendants, political donors and&nbsp;delinquent taxpayers.</li></ul>Of course&nbsp;this is all relative. While we can afford to be picky in Western countries, we might have to lower our standards in Africa or Asia.Fourth, we have improved our geo system. Each location - whether a city, region or country - has a unique ID, whose design reflects the hierarchical geo chain. We now know just by looking at their IDs that a person born in Boston and an homonym born in Massachusetts can actually be the same person.Last, we’ve put&nbsp;in place&nbsp;stringent mechanisms to detect typos. In the past, when confronted with a “Frederack Smith”, we would have thought: “We&nbsp;don’t know this first name. Let’s add it to our list and import this person.” We now hold on to the record, until we encounter more Frederacks.&nbsp;Past a certain number of occurrences, we create the records that were on hold. Otherwise, we delete them. This system has helped us identify scores of bad records, with first names such as “MichaelT” or “CarolM” that&nbsp;were in fact “Michael T” and “Carol M”.The first three countries we plan to import&nbsp;are the United States, the United Kingdom and France.We will also announce new important functionalities soon.Thank you for your&nbsp;continued&nbsp;support. We once compared the Population Project to a marathon. It looks more and more like some insane ultra-running race!

Three months ago, realizing we had way too many duplicates in our database, we stopped importing lists to take a closer look at our algorithms. We now believe we have fixed the most glaring problems. After wiping out the entire database, we’ve started reimporting US records.

We've started reimporting records

thumbnail_antoine-bello-avatar.jpeg

antoine-bello-avatar.jpeg

Among the many difficulties we face at the Population Project, none carry higher stakes than names. We’re not talking about the complexity induced by multiple alphabets or the relative scarcity of Korean names that makes for an inordinate number of homonyms. Those are future sources of worry as we’re only tackling Roman-alphabet countries for the moment. Yet, who would have thought Brazilian full names would pose so many problems?Brazilians typically have two first and two last names, some of which can be composed as in “Guilherme João Paulo Villas Boas Oliveira dos Santos”. To make matters worse, the lists we find are often stripped of their special characters such as hyphens or tildes. Out of the 150 million Brazilian full names we have processed, none was broken down, as is customary in other Latin American countries.This leaves us with an alternative: store the full name in one string (not ideal from a search perspective) or try to break it down in two or in four elements. We chose the latter option, only to discover it was more challenging than it looked as scores of Brazilian names can be both first and last names! When in doubt, we rely on frequency tables. For instance, “Braga” is 2,000 times more likely to be a last name, while “Juvita” is 100 times more likely to be a first name. If the ratio is less than 10:1, we skip the record altogether. We ALWAYS err on the side of caution.And of course, Brazil, as all countries, has its outliers, first names such as “Al Capone”, “Michael Jackson” or “Lady Di” that should in no way be separated. Well, at the risk of disappointing, expect a few mistakes on that front.

A day at the Population Project: Brazilian names

In the past three months, we’ve imported over 350 million records in the database of the Population Project. In the next few days, we will wipe out all of them and start from scratch.Why? Because we have too many duplicates, which is a sure sign that we have gaps in our filtering algorithm.Some duplicates are inevitable. The “Jane Simmons born on 2/7/1983” got married and became “Jane Milbanks born on 2/7/1983”. The matching date of birth is not enough to infer that the two Janes are the same person. Not surprisingly, women account for 53% of our US records, when the real percentage is closer to 51%.Other common sources of duplicates include typos (Paul Summers, born 5/7/1997 and Paul Summers, born 7/5/1997 are only separated by a date-keeping convention), name changes, truncated columns (shrinking Paul Summers’ name to “Paul Summ”). We’ve always been aware of these pitfalls, and while we do everything to address them, we know they will probably account for 1 to 2% of all records, until we develop more sophisticated tools to spot them.The problem right now is we have way more than the expected 3 to 5%. We’re already tracking at 102% of the US population, even though we have very few records of people under 18. This clearly indicates that our “disambiguation algorithm,” as we pompously call it, is too porous. As a reminder, we try to match every new record we import with one already existing in the database. Ideally, the new record will corroborate the old one or even enrich it with another datapoint.While reviewing our code one more time, we noticed a few gaps. Depending on which record came first, “John K Smith” and “John Kendrick Smith” were not always deemed compatible. To take another example, two “Mary Louise Miller” were not recognized as the same person, because the first one was labeled as “Female” (as per the list from which we extracted her information), while the other was only presumed to be “Female” (based on her first name).Once we’re done plugging all these holes, we will wipe out our data and start afresh, which should take less time than hunting for duplicates among the existing records.This is clearly disappointing. We’re working very hard to give the world its first population register but we’re only human. We’ll make more mistakes along the way. Rest assured though that we’re learning from each of them and that our enthusiasm has never been stronger.

medium_d8a3008db114d9cece22d651f9bcfc266771ed16-910x190.webp

small_d8a3008db114d9cece22d651f9bcfc266771ed16-910x190.webp

thumbnail_d8a3008db114d9cece22d651f9bcfc266771ed16-910x190.webp

d8a3008db114d9cece22d651f9bcfc266771ed16-910x190.webp

They say "Learn from your mistakes"

<a target="_blank" rel="" href="https://usbeketrica.com/fr">Usbek &amp; Rica</a> claim to be "a media questioning the future". Their article <a target="_blank" rel="" href="https://usbeketrica.com/fr">here</a>

First article about the Population Project in the French media

We were initially shooting for September but several issues arose during the summer, some mundane (hosting, mirroring…), some more substantial such as last-minute reforms of the way we store dates and places. When it recently became obvious that we wouldn’t be able to launch soon with all the bells and whistles we had in mind, we decided to ship a barebones version that would give the world a first glimpse of what we’re trying to accomplish. This initial version, online since October 5, contains only US data, which we’re adding to the tune of 20-30 million records a day. At the moment, records can be viewed but not edited. Plenty of stats are already available though, from the most popular first names to the age pyramid to the average completion of our records (100 means a full record, we currently stand at 60). So what are these other features in the hopper and when can you expect to see them?<ul><li>more countries will be added, starting with (but not limited to) Canada, the United Kingdom, and France.</li><li>all records will be editable, a la wikipedia. No account will be necessary.</li><li>registered users will be able to create records.</li><li>registered users will be able to tag records. Categories will include: “Me”, “Family”, “Friends”, and “Others”.</li><li>users not finding a record will be able to broaden their search to sound-like names and countries speaking the same language.</li><li>after confirming their identity, a user will be able to request their record to be taken down, in compliance with GDPR rules.</li></ul>We will introduce the new features gradually, hopefully by the end of the year. In the meantime, you’ll probably notice some bugs on the site. Be kind enough to report them <a target="_blank" rel="" href="https://thepopulationproject.org/contact-us">here</a>. We’re excited and proud to present this first glimpse of what the Population Project can be. Imagine our homepage with 4 billion records, 200+ countries covered and thousands of edits a day. It’s our vision, we hope it becomes yours.

medium_fdad59dcac2c6158b34e231a78f463d94de0fcb4-2892x1668.webp

small_fdad59dcac2c6158b34e231a78f463d94de0fcb4-2892x1668.webp

thumbnail_fdad59dcac2c6158b34e231a78f463d94de0fcb4-2892x1668.webp

fdad59dcac2c6158b34e231a78f463d94de0fcb4-2892x1668.webp

It’s a historic day: our site is up!

We get asked this question all the time. Why would you collect the names and birth information of all humans on earth?Simply put, because every life deserves to be acknowledged.It is estimated that one billion people - 15% of the world’s population - has no legal existence whatsoever. They cannot vote, travel, open a bank account. The Population Project might not reverse the legal plight of these people but it can give them that doesn’t cost much but means a lot: dignity.Besides that, ID or not, in a generation or two, most of us will be forgotten. The Population Project will keep a log of who lived when. It will be a resource for historians and genealogists alike. No record will ever be deleted.Last but not least, in a time when the human race is fighting for its survival, doesn’t it seem odd that we don’t know the name of our neighbors on this planet?

Why The Population Project?

With a team of 20 people scattered in nine different countries, the Population Project is growing rapidly. We are organized in three units:<ul><li>Programmers build the database that will one day host several billions of records. They encounter - and solve - new problems every day: how to automatically clean full names from various prefixes and suffixes (Jr, PhD, Mme…); until what age can we assume that a woman’s last name is also her maiden name; should we allow initials as first names, etc. Our team is based in the US and Canada and develops on <a target="_blank" rel="" href="https://us.4d.com/">4D</a>, the full-stack solution we have selected to power us.</li><li>Sourcers are scattered around the world. They look for lists in all countries using the Latin alphabet, i.e. pretty much the entire world minus Asia and Eastern Europe. To date, we have made strong to very strong progress in the US (our best territory), Costa Rica, the UK, France, Italy, Brazil, Mexico, Paraguay, Chile, Venezuela, Cameroun, Tanzania, Mali, Burkina Faso. We have collected at least some data in about 100 countries.</li><li>Processors are based in Madagascar. They sort out and clean up the lists uploaded by sourcers. Experts in Excel, macros and text formats, they ensure the highest level of data quality.</li></ul>Modern non-profits use the same communications tools as startups: <a target="_blank" rel="" href="https://slack.com/">Slack</a>, <a target="_blank" rel="" href="https://zoom.us/">Zoom</a>, <a target="_blank" rel="" href="https://airtable.com/">Airtable</a>, <a target="_blank" rel="" href="https://www.notion.so/">Notion</a>, <a target="_blank" rel="" href="https://www.dropbox.com/">Dropbox</a>, <a target="_blank" rel="" href="https://wise.com/">Wise</a> and many other dedicated tools. Except in Madagascar, there are no schedules: everybody makes their own hours. We all strongly believe in our mission. At first, the challenge of capturing the names and dates of birth of all humans alive can seem totally crazy. But we have a secret weapon: time. We only need to encounter a name once - at a <a target="_blank" rel="" href="https://www.syracuse.com/schools/2021/06/graduation-2021-see-list-of-oneida-high-school-graduates.html">graduation ceremony</a>, in the <a target="_blank" rel="" href="https://results.nyrr.org/event/M2021/finishers">New York marathon results</a> or on a <a target="_blank" rel="" href="http://transparencia.recife.pe.gov.br/codigos/web/covid19vacinados/vacinados.php">Brazilian official vaccination list</a> - to add it to our database where it will stay forever. This is why even in hard times (and there are some) we remain confident: if we haven’t found your name yet, we will - in time.

Time is on our side

No news from the Population Project doesn’t mean bad news. Quite the opposite actually: it means that, like the little engine that could, we’re firing on all cylinders!Our team of list sourcers now collects an average of 1 million names per day, weekends included. Lists mostly come from South America and Africa (we're limiting ourselves to the roman alphabet for the moment).Our biggest contributors: social programs in Mexico, vaccination reports in Brazil, electoral rolls in Ivory Coast, exam results in Mali and Cameroon. When we find a good list, we dig previous years’ archives and make a mental note to check back next year. It’s too early to compute ratios but it’s safe to say that a fair amount of lists are fully renewed every year (think exam results or high school graduations).Collecting all these names would be pointless without our expert staff of processors. Based in Madagascar, they evaluate the reliability of the source, clean up the files, remove unnecessary columns and duplicate records, and save the resulting spreadsheets in a giant Dropbox (see below).<img src="https://tpp-staging-pat.s3.eu-central-1.amazonaws.com/2dba9bfb2803ca2567e96ffd85230ed0f90aa77a_2446x834_4d45f549ff.jpeg" alt="2dba9bfb2803ca2567e96ffd85230ed0f90aa77a-2446x834.jpeg">We’re also moving on the IT front, building the 4D database that will host all this data and, ultimately, power our website. Now that we’ve settled on an <a target="_blank" rel="" href="https://en.wikipedia.org/wiki/Hierarchical_model%E2%80%93view%E2%80%93controller">HMVC model</a> (see previous post), we work on three separate but interconnected bases:<ul><li>Background (where we host all the name knowledge we’ve amassed about every country);</li><li>Import (where we’ll process the lists with a view to separating new humans from returning ones);</li><li>Repository (where all records will be stored and available for query).</li></ul>We work a lot but no one complains! Knowing we’re in it for the long haul helps. Adding 1 million names a day as we do now (and assuming all of them are new, which they’re not) means it would take us over 20 years to complete our mission. Let’s see if we can do better than that…

The little engine that could

Basic workflowVolunteers or freelancers collect lists of humans around the world. Some of these lists are several millions long (i.e. registered voters in the state of New York), others only a few hundred. Some lists contain all the fields we’re looking for (i.e. first name, middle name, last name, maiden name, sex, country, date of birth, place of birth), others just contain bundled names (“John Smith” or “John T Smith”).We clean the lists (converting files from PDF to Excel, getting rid of the unnecessary columns, removing duplicates). Files are then ready to be imported. Imports are called jobs. Depending on their origin, lists are labeled as official, semi-official or informal. The level of certainty of each field is set accordingly. All lists are assigned a nationality. For instance, “List of graduates from Harvard Business School” is labeled “USA”. “List of public employees from the state of Yucatan, Mexico”, is labeled “MEX”. A few lists, such as “Olympic medal winners” are labeled “INT” (for “International”). In this case, each human on the list has their nationality clearly specified. Under no circumstances can a human be imported without a country.Logic of the structureWe will operate on the HMVC (Hierarchical Model View Controller) model. Basic tasks will be broken down between several bases linked together.<img src="https://tpp-staging-pat.s3.eu-central-1.amazonaws.com/471a13552100dc3c77e848c9225c65aea4d3619c_468x259_75f7dc9e68.png" alt="471a13552100dc3c77e848c9225c65aea4d3619c-468x259.png">Bases will include, among others:<ul><li>a background base encompassing all low-variability data such as countries, first names, last names, language-related info, etc.</li><li>an ingestion base, which will work as a chamber between excel files and the main repository.</li><li>a main repository, which will contain all human records organized by countries for small countries, and by countries/alphabetical tables for larger countries. If or when the main repository becomes too large it can always be broken down into several bases after a country/continent logic.</li><li>a statistics base, which will periodically query the main repository to compute and store statistics.</li></ul>Logic of import jobsa. Selecting the import fileWe will put controls in place in order to limit the ingestion of junk.<ul><li>make sure there is not text in the numeric columns and vice-versa;</li><li>make sure by sampling that the first and last name columns haven’t been mixed up;</li><li>raise an alert when too many first or last names from a list can’t be found on Forebears, etc.</li><li>other tests to ensure that we never import a defective file.</li></ul>b. Logging the data (ingestion base)Each human on the list is processed separately and goes through several steps:<ul><li>Names are cleaned (to get rid of spaces, periods, etc.).</li><li>As names are often bundled, we separate them using a proprietary algorithm. For instance, “Bundled name : De la Tour Pierre Jacques Henri” becomes Last name: De la Tour; First name : Pierre; Middle name : Jacques Henri.</li><li>If the nationality of the human is unknown, we give it the overall nationality of the list. Our concept of nationality is intentionally loose to account for people who have lived in several countries. A same individual can be listed under two or three different countries. We might want to remedy that at some point, but for the time being, we think it’s not a major issue.</li><li>If we ignore the sex of a human, we try to guess it based on their first name and country. We only assign a gender if it’s more than 95% certain in <a target="_blank" rel="" href="http://Forebear.io">Forebear.io</a>’s API.</li><li>We register the year, month, and day of birth if we have them (three separate fields). The year of birth can sometimes be estimated from the nature of the list. For instance, a French student taking the Baccalauréat has a 95% likeliness to be between 17 and 21. In that case, we log the year corresponding to an age of 19, with a confidence margin of 2 years (in a separate field).</li><li>We register the city, region, country of birth when we have them (rare).</li></ul>c. Computing additional information (ingestion base)<ul><li>We log what we call the full name (last name + ”_” + first name), for instance “Smith_John” or “De la Tour_Pierre”.</li><li>We transform each name (first, middle, last, maiden) into a soundex or metaphone, which will greatly expand the functionalities of the search. Johnson_Michael becomes J525_M240 (we’re using the most basic soundex as an example, we might resort to a more sophisticated one).</li><li>We elaborate on the soundex to calculate a hash that takes into account the other information we have about Michael Johnson. We use a smart hash that prioritizes the most important/discriminating information by placing them earlier in the string. For instance, we are much more likely to know the human’s birth year but not the day than the other way round.</li><li>We compute three individual indexes, comprised between 1 and 100 on a logarithmic scale.</li></ul>o The basic index expresses how unique a human is, first based on their first and last name. Michael Johnson for instance has a 90+ basic index. To avoid querying the “background” base incessantly when importing, for instance, American names, we’ll store US first and last names in an array in the ingestion base for the duration of the import. o The full index expresses how unique a human is, based on all the information we possess. Because we know a lot about him, Michael T Johnson, born on April 24, 1973, in Atlanta, GA has a full index of 1 (1 symbolically expressing the unicity of each human being). o The completion index expresses how much information we have on an individual. Each field adds a certain number of points, up to 100 (when we know all there’s to know).d. Comparing the human’s soundex with the soundex of humans already in the base (ingestion base)The ingestion base will contain one table per country with all the distinct soundex pairs. When dealing with human “Michael Johnson”, we’ll look in the Soundex_pairs_USA table whether the corresponding soundex, “J525_M240”, is new or not.If it is, we add “J525_M240” to the base Soundex_pairs_USA and we add the property “new” to our object “human”. If if isn’t, we give the object human the property “test”. Then the object is placed into a temporary collection (or into two temporary collections: one for all .new items and one for the .test items).e. Creating new humans or enriching existing ones (main repository)Every 1,000 records, we send the collection(s) to the main repository.For lack of testing, we’ll assume for the moment that Humans tables will host up to 25 million records (roughly the population of Australia). Countries smaller than Australia will fit in one table. The US will require approximately 14 tables, China and India 50 or 60 tables each. It might make sense to regroup small countries according to a geographical logic (Caribbean countries, Baltic countries, Pacific islands, etc.).In countries over 25 million people, humans will be dispatched to a specific table based on the initial of their first name. In the US, for instance, the tables will go Humans_USA_BED, Humans_USA_CAF… all the way until Humans_USA_ZZZ. Australia’s only table will be Humans_AUS_ZZZ.Humans with the property “new” are immediately created in the table corresponding to their name, while humans with the “test” property need to be compared to existing records. Let’s say there are 7,830 humans with the soundex pair J525_M240. We look within this selection how many humans share the same hash as our Michael Johnson. If there are none, we create our record. If there are one or more, we have no other choice than to compare the records one by one.In so doing, we go by a basic principle: we only create a human if he/she can be differentiated from all other humans in the base on at least one feature. We won’t import another John Smith born in 1983 if we already have a John Randolph Smith born in 1983 (because they could be the same person). On the contrary, we will import a John T Smith born in 1983 (as John T Smith and John Randolph Smith are clearly different individuals).When in doubt, we err on the side of caution. The only instances where we “merge” two humans is when their basic index makes us comfortable enough. A Michael Johnson born on 3/25/70 and another Michael Johnson born on 3/25/71 are very likely to be two different persons since more than one Michael Johnson is born every day in the USA. On the other hand, if we have two Nanshapur da Oliveira in Italy, one born on 3/25/70 and the other born on an unknown date in 1970, they’re very likely the same person. The various thresholds used in the algorithm need to be finetuned.Last rule: corroborating a bit of information increases the level of certainty of the corresponding field, using a simple set of rules that can probably be improved:<ul><li>If the old information (can be a maiden name, a year of birth, etc.) has a lower certainty that the new information, we take the certainty of the new information.</li><li>If the old information is more certain than the new information, we keep the certainty of the old information.</li><li>If old and new information are equally certain, we increase the certainty according to the following formula: Increased Certainty = 1- ((1 – Old Certainty)*0.85)). For instance, two 80% sources result in an 83% certainty.</li></ul>Structure of the basesa. Background baseThe background base includes the following tables:<ul><li>One country table, listing for each country its ISO-3 code, population, etc.</li><li>One first_name_sex table per country. Each table contains between 1 000 and 30 000 common first names of the country + the gender breakdown. Necessary to calculate the basic index. Syntax: First_name_sex_FRA.</li><li>One last_name table per country. Each table contains between 1 000 and 20 000 common last names of the country. Necessary to calculate the basic index. Syntax: Last_name_FRA.</li><li>Two tables to store the results of all the Forebears API queries we perform (so that we don’t run the same queries over and over).</li></ul>b. Ingestion base<ul><li>One import table, which might not even be necessary as records could be pass-through and stored only temporarily.</li><li>One job table to record imports and all metrics associated with them (number of humans processed, created, merged, etc.).</li></ul>c. Main repositoryHumans tables contain seven fields. Whether all of them need to be indexed remains to be decided.<ul><li>the human id;</li><li>the job id (through which job was this human imported?);</li><li>a name object. Attributes include: first, first_certainty, first_metaphone; middle, middle_certainty, middle_metaphone; last, last_certainty, last_metaphone; maiden, maiden_certainty, maiden_metaphone.</li><li>a sex object. Attributes are sex (1 or 2) and certainty.</li><li>a birth object. Attributes include : day, day_certainty; month, month_certainty; year, year_certainty, year_confidence_margin; city, city_certainty; region, region_certainty; country, country_certainty.</li><li>a death object. Same attributes as birth.</li><li>a human_index object. Attributes (all three integers between 1 and 100) are: basic index; full index; completion.</li></ul>Birth and death years can be expressed as intervals (1970 with a confidence margin of 2 years means “between 1968 and 1972”. This is especially useful when the year of birth can be estimated (for instance, people finishing high school in the US are most likely between 17 and 21 years old).The main repository will have to be broken down into several bases. Because a 4D base can only sort in one alphabet, we’ll need dedicated bases for China, Japan, Thailand, Korea, etc.Some data will be used much less frequently than others (for instance all death-related info). We might want to store it in another base. As an example, Australia would have two “Humans” bases, one with the basic info (names, birth, sex, etc.) and another for the death info. In the beginning, the “Death” base would be almost empty and would fill up extremely slowly. Or the first base would be only for live people and at the time of their death, they would migrate to the “Dead people” base. The merits of these ideas need to be further assessed.d. Statistics baseWorkers will query the main repository at regular intervals to calculate various indicators.<ul><li>Level of completion of humans per country (as an example, we have few records in Cameroon but they’re almost complete, lots or records in Mexico but they’re almost empty).</li><li>Most popular last names, per country and overall.</li><li>Most popular first names per gender, per country and overall.</li><li>Average age per country.</li><li>Gender breakdown by country (women tend to be underrepresented in many countries’ lists).</li></ul>Some of the most interesting stats will pertain to the Population Project itself.<ul><li>Percentage of humans ingested that were already present in the base (will decrease over time).</li><li>Number of humans per country, as a number and as a percentage of the country’s population.</li><li>Number of humans added in the last hour/day/week/month.</li><li>Number of lists submitted by the volunteers.</li><li>Titles of the most recently submitted lists, etc.</li><li>Backlog of lists and humans to process, etc.</li></ul>Results will be stored in “historic tables”, making it possible to calculate variations and display graphs.Search functionality (website)Because we’ve chosen to organize humans by country, users of the website will be strongly encouraged to select a country prior to running a query. One way could be to preselect the country corresponding to their IP address in the “Country” dropdown menu.If a query doesn’t return any results for a specific country, we’ll offer to extend it to neighboring countries (e. g. South American countries for Argentina) or to the entire world. One could imagine the following message. “Your query for Luigi Amekporphor didn’t yield any results. Would you like to extend it to other European countries or to the rest of the world (will take longer)?”.From our perspective, searching for one name in the entire world means querying 239 bases, a number that we think can be reduced to 100 or so by regrouping small countries.The use of metaphones will allow us to display extensive results, returning for instances the various spellings of Jansen or Mohamed.In the long run, we might give researchers the possibility to build and download (anonymized or not) datasets.Challenges and constraints<ul><li>Performance. Should all fields be indexed? Can the dialogue between the import base and the repository be fast enough to accommodate the ingestion of millions of records a day?</li><li>Preemptive programming so we can run multiple import jobs in parallel.</li><li>Handling various alphabets, each requiring their own database for sorting purposes.</li><li>Working with multiple developers. The HMVC structure of the project should help.</li><li>Create workers and use artificial intelligence to sweep the data at regular intervals for potential mistakes, typos, duplicates, etc. Over time, the names of places will need to be harmonized (London, Londres, Londra, Londen, etc. should be “London” ).</li><li>24/7 availability of the site, irrespective of maintenance.</li></ul>Next stepsa. Phase I<ul><li>Build the import process. This will force us to: settle on a field structure, choose a soundex method, program the hash, etc.</li><li>Start work on the website.</li></ul>b. Phase II<ul><li>Trial imports to test key assumptions such as number of humans per table, ideal size of import batches, etc.</li></ul>c. Phase III<ul><li>Build the stats base: build ad hoc workers to query the main repository; archive historic results; implement visualization tools.</li></ul>

A not-too-technical explanation on how the Population Project's database will be structured.

Memo on the Population Project's database structure

It has been a busy summer at the Population Project. After the initial sprint, we’re settling into a marathon pace more suited to our time horizon, which is measured in years, if not decades. We have more clarity on the challenges ahead:Gathering lists is relatively easy in some countries (the US, France), much more difficult in others (Germany, Peru). Some countries have the good sense of separating first, middle, and last names, while others only display a full name that is sometimes difficult to break down. Hiring and coordinating volunteers has proved trying, leading us to target fewer countries where we hire highly productive freelance web researchers. It’s still a bit early to discuss metrics as numbers vary greatly from one week to the next. We can find the names of 20 million Mexican public employees on Monday and collect only 500,000 names the rest of the week. Overall though, we think we’ve only tapped the tip of the proverbial iceberg.Processing lists is less rewarding than finding them but just as important. Most lists come in PDF format. They need to be converted and purged from any unnecessary or redondant information - a process that take anywhere between two minutes and one hour per list. As we can’t keep up with all the lists we receive, we’ve decided to outsource part of our workflow to a BPO (business process outsourcing) firm based in the Philippines and Madagascar. The first tests will begin next week. Being able to count on an apt and experienced partner should definitely free up some of our resources.IT remains the biggest obstacle we face. The size of the database we plan on building doesn’t allow for shortcuts. Slight conception mistakes might prove absolutely catastrophic when we have one or five billion records. After an initial development in Postgres, we’ve scrapped our entire code and started anew on <a target="_blank" rel="" href="https://us.4d.com/">4D</a>, a powerful yet agile relational database system. The problems (comparing humans, merging records, etc.) haven’t gone away but they’re easier to frame. We won’t publish a beta version of the site as originally planned but hope to have a prototype some time during the first half of 2022.As always, we welcome your feedback. If you feel like helping in any capacity - be it to collect lists, process them, or design our website - please <a target="_blank" rel="" href="mailto:antoine@thepopulationproject.org">contact me</a>. One last ask: please share this post or your LinkedIn page or Facebook page - we haven’t had time to hire a PR firm yet!

It has been a busy summer

<h3 style="text-align: start">Where we stand - July 2021</h3>Two months after its inception, it seems a good time to assess the first steps of the Population Project. We owe it to ourselves and to our volunteers to honestly review the good and the not so good if we want to have a shot at executing our gigantic task.The goodThere have been plenty of reasons to rejoice:<ul><li>The warm response to the project is the most important of all. From every corner of the world, we have heard support for our initiative. Granted, the naysayers probably didn’t bother coming forward but we can safely say that our vision is resonating with many.</li><li>We are gradually honing our message. When we launched the Population Project, we faced the same question over and over again: why list the name and date of birth of all humans alive? Today we have better answers to offer: to acknowledge every life; to bear testimony of who lived when; to give each earthling a chance to know their neighbors; last but not least, because it would be beautiful.</li><li>We’re tackling head-on the delicate issue of privacy. We have hired Privacy Interpol, a Berlin-based consulting firm that is currently crafting our privacy policy and making sure that we comply with all rules and regulations.</li><li>Our website works well. Its content management system makes it incredibly easy to update the <a target="_blank" rel="" href="https://thepopulationproject.org/guides/volunteer-handbook">Volunteer Handbook</a> or edit blog posts. It’s still a pain to find us on Google but we will get there.</li><li>A few incredible individuals (you know who you are) have proved that one motivated and skillful volunteer can source several dozens of lists a day. Some of them will contain 80 names, others 400,000. Some will be limited to first and last name, others will include date and place of birth. We take them all.</li><li>We have gathered about 300 millions profiles, mostly in the US, France and the UK, but also in India, Zambia, Algeria, Cameroon, Canada, the Philippines and many other countries.</li><li>We’re constantly streamlining the core process that goes from sourcing a list to creating an Excel file ready for import. Volunteers now seamlessly upload the URL of the lists they’ve found. A ticket is automatically created in a workflow management software, where anybody can follow the progression of the job (downloading of the file, conversion from PDF to Excel, cleaning of the Excel file, import in our database, etc.). As much as we’d love to rely on volunteers for all tasks, we’ve started hiring data-processing workers in developing countries.</li></ul>The not so goodThere are several key areas that haven’t lived up to our expectations.<ul><li>Our Postgres database is the matrix of the Population Project. Every time we feed it a name, Postgres determines whether it’s a new entry or another iteration of a profile already in store. If the latter, Postgres compares the records field by field (middle name, date of birth, etc.) to determine whether they should be merged or not. The difficulty is compounded by the fact that there can be 30,000 John Smiths and that we plan on ingesting several millions of records a day at scale. We’ve pretty much written the instructions but they’re still too slow to execute. If you know Postgres and are looking for a challenge, contact us by all means!</li><li>As we source lists from more and more countries, it’s becoming clear that we need a reliable method to break down full names into first, middle and last name. What is relatively easy in French or English turns out to be a nightmare in Portuguese and Arabic. This looks like a typical use case for machine learning but we don’t possess this skill yet.</li><li>We’re not sourcing enough lists, especially in developing countries. This is a shame because our ads generate a fair amount of interest. But many applicants seem to like the idea of volunteering more than the actual work it entails. They talk the talk where we’d like to see them walk the walk. Part of the responsibility undeniably lies with us (sourcing lists involved too much technical skills in the beginning), but we must instill an action-based culture among our recruits.</li><li>We don’t have specific strategies for India and China, which together account for more than a third of the world population.</li><li>Antoine can’t do it alone. We will soon announce crucial additions to the team.</li></ul>Asks – How you can helpYou can become a research volunteer, by sourcing lists of names in your country. More details in the <a target="_blank" rel="" href="https://thepopulationproject.org/guides/volunteer-handbook">Volunteer Handbook</a>.Not ready to commit? Anybody can submit one or several lists <a target="_blank" rel="" href="https://thepopulationproject.org/volunteer/submit">on our website</a>. It takes less than a minute.We’re looking for strong profiles in the following fields :<ul><li>Back-end (preferably Postgres);</li><li>Front-end, to prepare the second version of our website;</li><li>Machine learning;</li><li>Data science;</li><li>Scraping;</li><li>Search engine optimization.</li></ul>Whether you view your contribution as a one-time gig or you’re willing to contribute several hours a week, let’s talk!

We owe it to ourselves and to our volunteers to honestly review the good and the not so good if we want to have a shot at executing our gigantic task.

Where we stand - July 2021

After the US, we looked at France, a heavily centralized country that has recently drawn praise for migrating many governmental functions online. First, the bad news. Birth records archives are not public until after 75 years past people's birth or 25 years past their death. And unlike the US, voters lists are strictly confidential. There go our two biggest purveyors of names.Fortunately, France likes grades and exams. The results of the Baccalauréat and various other exams are widely published in the media and are not too difficult to scrape. Each record contains the full identity (first, middle and last names). We assume that the girls' name is also their maiden name and that 95% of candidates are between 17 and 21 (or, in our jargon, 19 with a confidence margin of 2 years).Only the last three years' results are available online. Hopefully the <a target="_blank" rel="" href="https://archive.org/">Internet Archive</a> has more in store. But this is a great start: we now have most of the age class 2000-2002.We quickly find two more administrative troves: the list of newly naturalized French citizens (complete with date and place of birth) and the list of candidates running for elections. Only 10 people or so run for President but about half a million were candidates in the 2020 municipal elections. That's almost 1% of the entire population!We continue our journey in the meanders of the French administrations. To advance their career, civil servants such as police officers, engineers and even custodians are encouraged to take internal exams, the results of which are available to the public.And, like in almost all countries, special professions such as doctors, attorneys or nurses keep lists of their members. The numbers quickly add up.French sports federations tend to be more transparent than their American counterparts. The table tennis federation ranks no less than 60,000 amateurs! Among other surprising hits are horseback riding, sailing, fencing, ski, swimming and track and field - all individual sports as might be expected.More disappointing is the absence of a database of civil award and military decoration recipients. No one seems really interested in giving veterans the recognition they deserve…All in all, France is much more a work in progress than the US, due to the lack of a major all-encompassing source. Yet we've found many reasons to believe.

Populating France

Six months ago, when we started to think about what would become the Population Project, we decided to begin our investigations with the two countries we know best: France and the United States. Granted, these countries can hardly be viewed as representative of the world population. Already highly digital, they could be expected to post more information online than the average country. But that was precisely the point: if we couldn’t quickly account for a significant proportion of the French and American population, then the notion of one day mapping India or the African continent was outright delusional.The US quickly proved to be a public data Eldorado. For starters, voters’ lists are available to the public in every state. Sometimes you can download them on the state or county electoral commission’s site, sometimes you have to order a disk. Each record contains the first and last name, the middle initial, the sex and the date (or only the year) of birth. In a little more than a month, we were sitting on 200 million records!It took us a bit longer to discover a second gold mine: lists of public employees along with their salaries and pensions. Some states go to great lengths to conceal their data but with a bit of tenacity, we were able to get ahold of most of it. Needless to say, we have no interest in how much civil servants make in Minnesota. We only store a handful of data points for each human and salary isn’t one of them.With something like 80% of the American adult population in hand, we shifted our efforts to smaller datasets. For instance, there are several million inmates and sex offenders in the US. Over a million students graduate from college every year, their names printed in universities commencement programs. This made us realize that if we didn’t know the graduate’s dates of birth, we could still guess their ages with a two or three-year confidence margin.We then branched into sports. Individual sports federations typically publish rankings of their discipline. Swimming, track &amp; field, ski proved the most fruitful avenues, with chess, bridge and Rubik’s cube (50,000 names!) not far behind.It became obvious at this point that younger generations would be the most difficult ones to come by. Except for 12-year-old swimmers and the occasional chess prodigy, we are desperately lacking in the under 18 category.Overall, we consider our American campaign an unmitigated success. It has left us with hundreds of millions of records, including lots of overlapping ones that enable us to corroborate data and to gradually enrich profiles.Next week: France!

How we quickly acquired 70% of the US population.

The Population Project Blog

Updates, thoughts, and reflections from the Population Project team

AB
Antoine Bello · April 12, 2023

AB
Antoine Bello · February 9, 2023

AB
Antoine Bello · January 9, 2023

AB
Antoine Bello · November 3, 2022

AB
Antoine Bello · October 6, 2022

AB
Antoine Bello · August 2, 2022

AB
Antoine Bello · March 4, 2022

AB
Antoine Bello · December 8, 2021

AB
Antoine Bello · October 6, 2021

AB
Antoine Bello · September 19, 2021

AB
Antoine Bello · July 12, 2021

AB
Antoine Bello · June 10, 2021

AB
Antoine Bello · June 6, 2021

The Population Project Blog

Updates, thoughts, and reflections from the Population Project team

ABAntoine Bello · April 12, 2023

ABAntoine Bello · February 9, 2023

ABAntoine Bello · January 9, 2023

ABAntoine Bello · November 3, 2022

ABAntoine Bello · October 6, 2022

ABAntoine Bello · August 2, 2022

ABAntoine Bello · March 4, 2022

ABAntoine Bello · December 8, 2021

ABAntoine Bello · October 6, 2021

ABAntoine Bello · September 19, 2021

ABAntoine Bello · July 12, 2021

ABAntoine Bello · June 10, 2021

ABAntoine Bello · June 6, 2021