Four staffers at the Associated Press shared a 2012 Pulitzer Prize on Monday for exposing the New York Police Department's clandestine spying that monitored daily life of Muslim communities. The Pulitzer board at Columbia University in New York said the AP series resulted in "congressional calls for a federal investigation, and a debate over the proper role of domestic intelligence gathering." The journalists are Matt Apuzzo, Adam Goldman, Eileen Sullivan and Chris Hawley. The AP is a non-profit news cooperative owned by U.S. newspapers.

The Seattle Times also was honored in the investigative reporting category for articles showing "how a little-known governmental body in Washington State moved vulnerable patients from safer pain-control medication to methadone, a cheaper but more dangerous drug, coverage that prompted statewide health warnings." The journalists are Michael J. Berens and Ken Armstrong. You can read that series here.

Highlights of the AP investigation are here. A summary:

Domestic spying
"AP's investigation has revealed that the NYPD dispatched undercover officers into minority neighborhoods as part of a human mapping program. Police also used informants, known as "mosque crawlers," to monitor sermons, even when there was no evidence of wrongdoing. The articles showed that police systemically listened in on sermons, hung out at cafes and other public places, infiltrated colleges and photographed law-abiding residents as part of a broad effort to prevent terrorist attacks.

"Individuals and groups were monitored even when there was no evidence they were linked to terrorism.

---

"The AP also determined that police subjected entire neighborhoods to surveillance and scrutiny, often because of the ethnicity of the residents, not because of any accusations of crimes. Hundreds of mosques and Muslim student groups were investigated and dozens were infiltrated. Many of these operations were built with help from the CIA, which is prohibited from spying on Americans but was instrumental in transforming the NYPD's intelligence unit after 9/11."

Reporter Apuzo describes the reporting on the series in a podcast for Pro Publica, the nonprofit investigative news organization. You can listen to the podcast here.

More winners
The winners in journalism, letters and the arts are listed at the Pulitzer Prizes site at Columbia University, and nominated finalists who did not win are listed separately.

The Huffington Post news website won its first Pulitzer Prize, for national reporting, for articles describing wounds suffered by American veterans in Iraq and Afghanistan. The series is called "Beyond the Battlefield."

 

Who says there's no free lunch?

You may have read over the past week about the release of 1940 Census records on a new U.S. government website, a site that buckled under the huge demand from people looking up details on the lives of their friends and relatives from the Great Depression.

You may not have realized that the site was built for U.S. taxpayers for the price of — not one dime. A company from Silicon Valley built the site, and is operating it, for free. Genealogy buffs have been using the site for a week now to check millions of records. (See our earlier story for tips on searching the 1940 Census, and examples of people who have found relatives.)

Of course, the company, Inflection LLC of Redwood City, Calif., did get something in return for its effort: a free copy of those 3.8 million images of records from the 1940 Census. While other companies paid $200,000 for a set of the public records, Inflection can use those records in its for-profit business, a genealogy site called Archives.com.

It's a barter system for federal records: the public gets a free official U.S. website, and the company gets free data. It's been done before, as when the U.S. Patent and Trademark Office gave data to Google, which since 2006 has hosted the site for free as Google Patents.

---

Do you approve of the approach that the National Archives took, giving the data away in exchange for the free website? And what stories have you found in the 1940 Census? Add your story in the comments below or on our Open Channel page on Facebook.

Inflection also was hoping to get a boost to its reputation for building websites that could withstand a storm of traffic.

Performance standards in the contract
Both the company and the National Archives and Records Administration (NARA) had anticipated that the site would draw a crowd, as 72-year privacy restrictions expired and the records became available. What happened next lends credence to the boast that genealogy is the country's favorite hobby.

The contract says, "Drawing from NARA's experience in releasing the 1930 Census, and the experience of the National Archives of the United Kingdom when they released their 1901 and 1911 Censuses, NARA anticipates immense interest in the 1940 Census and a tremendous increase in traffic to its www.archives.gov web site." (Here's the contract in a PDF file.)

But how much of a crowd?

Here are the performance standards in the contract:

• "When browsing from one image to another, each image should be presented to the user in 3 seconds or less."
• "When moving from the standard rendered image to each zoom level (e.g. zoom 1x, 2x, 3x), the reformatted image should be rendered in 2 seconds or less."
• "Support up to 10 million hits per day while providing response times of less than three seconds for keyword searches of the descriptive metadata."
• "Support up to 25,000 concurrent users."

There was one more element in the contract, a somewhat vague requirement that Inflection increase service if demand was greater than anticipated.

• "Scale on demand in the event that 10 million hits and/or 25,000 concurrent users are exceeded to ensure that the performance requirements ... are still achieved."

The crowd certainly exceeded those levels, as the most old-fashioned sounding search term possible, "1940 Census," became a top "trending topic" on Google and Twitter.

Most people seemed to get little or nothing from the site on the first day, including Census leaders, who were prepared to show off how easy it was to look up their grandparents. When the site stuck on "loading image," as it did for many other users, the officials resorted to showing a PowerPoint presentation with the results from an earlier search.

A 'tsunami'
As Inflection's general manager, Joe Godfrey, told us last week, "We were expecting a flood, but we got a tsunami."

• On Day One, Monday, an estimated 100 million hits, or requests, with 22.5 million hits in just the first three hours. Though Inflection scrambled to improve service, the site was unusable for many users on the first day. The company added more servers through Amazon Simple Storage Service, its cloud data service provider, and also restricted some features on the site (such as zooming of images), until finally it was able to get on top of the traffic.
• On Day Two, Tuesday, the numbers haven't been totaled, but it's believed to be higher than on Day One, with an estimated 40.1 million hits in the three-hour peak.
• By Friday, the site was stable with about 60 million hits per day, and had served up more than 80 million images, or about 61 terabytes of data, the National Archives said. (That's more than the data contained in the first 20 years of astronomical observations by the Hubble Space Telescope.) The service quality was better than called for in the contract, with a load time of about 1.8 seconds per page, according to the Archives.

In other words, this might have been a good project for a "soft launch."

The contract called for extensive load testing before the release. We asked the National Archives for copies of those test results, but its spokeswoman said it wouldn't be able to provide them. But it said the site was tested to handle more than 70,000 simultaneous users — more than the contract called for, and fewer than the level that resulted.

A 'no-cost contract'
No-cost contracts are allowed under Federal Acquisition Regulation competitive procedures. This contract has a one-year base period and options to extend for four more one-year periods.

"NARA provided a copy of the data to Inflection at no cost, copies that were sold to others for $200K," said spokeswoman Laura Diachenko of the National Archives. "Why Inflection agreed to this is a better question for them, but we are very happy to have them as a partner. They have experience with Census data, and managing access to large data sets, the capabilities we were seeking for this project."

She added, "Even though this is called a no-cost contract, the Government did incur costs — in this case, aside from our resources, we also provided a copy of the 1940 Census to Inflection, at no cost.  In this particular case, we provided them data that they wanted in exchange for hosting access to this data.  Their interest was in getting the data (for their archives.com business), and for business development (attracting users to their site and eventually converting them to a subscriber."

Inflection's Godfrey said, "The primary value for us was in building our brand/notoriety, leveraging and expanding our technical expertise/infrastructure and helping to getting this extremely valuable record collection into the hands of as many people as possible.  Also, our engineering team (like all great engineers) are motivated by tackling challenging technical problems, and so the team was very excited to work on this."

Competition
All or most of the 1940 Census is now available free from several other companies, which had to pay for the public records. As a sort of loss leader, other genealogy sites, even the commercial ones, are making the 1940 Census records available for free, to subscribers and non-subscribers alike.

Here's how the race worked: All the commercial sites that chose to buy the data for $200,000 were handed a rack of hard drives full of 20 terabytes of images, taken from 4,745 rolls of microfilm, at 12:01 a.m. on April 2, or 72 years and a day after the Census Day in 1940.

By Thursday, a relatively new genealogy site called myHeritage, was the first to have all the images online. Also making images available for free are Ancestry.com, a commercial site, and FamilySearch.org, owned by the Church of Jesus Christ of Latter-day Saints.

Thousands of volunteers are working on the next step: indexing the records by name, just as previous Census releases have been indexed by volunteers. Until those indexes are finished, searching is done only by address or neighborhood.

Your view
Do you approve of the approach that the National Archives took, giving the data away in exchange for the free website? And what stories have you found in the 1940 Census? Add your story in the comments below or on our Open Channel page on Facebook. See our earlier story for tips on searching the 1940 Census.

Update, 5:40 p.m. ET: The firm at the center of today's census records meltdown says, "We were expecting a flood, but we got a tsunami."

"We had estimates of how much traffic was going to hit the site, and we did performance testing at several levels above that, but we were surprised by the traffic," Joe Godfrey, senior director of product and general manager for Inflection, a Silicon Valley database company."

Inflection was hired by the National Archives and Records Administration, which provided the 1940 census records. Inflection buiilt the search engine to serve up the records, and relied on Amazon Simple Storage Service (Amazon S3) as the cloud service provider. Inflection has been adding more of a pipeline to Amazon all day, adding the ability for more simultaneous connections, but so far searches for census records are running slowly or not running at all for many users.

---

The company is trying to serve up 3.8 million images of census documents, each with multiple views at different zoom levels, with each file being 10 megabytes or larger.

Godfrey said the situation has improved, and engineers are hoping by the end of today to have the situation squared away.

Earlier:

Embarrassed by a computer system that crumbled under public demand, the National Archives and Records Administration said Monday that it's working to add more servers for the release of 1940 Census records. For more users the wait to see records on family members from the Great Depression era will go on for a while longer.

The Archives had hired Inflection, a Silicon Valley database company, to run the computers, but frustrated users lit up Facebook and Twitter with complaints about images that were said to be "loading" but never arrived.

"Our testing indicated NARA and Inflection could handle the load, but 1.9 mil visitors caused issues we're working to resolve," the Archives said via Twitter. Later it added, "We'll let you know as soon as we have another update - thank you for your patience, we know it's incredibly frustrating."

Even agency officials, during the webcast to kick off the day, couldn't get images to load when they tried to look up their own relatives.

In Springfield, Ohio, Facebook user Val Lough commented on our page: "It's very sweet of them to put all of these records on line. It would be even nicer of them to make the records VISIBLE. None of them will download, I have a browser window opening that's 'loading' the documents and has been for about 20 minutes. You might want to find out what their issues are. It would be faster to mail a public records request to the National Archives." Many others are tweeting about delays.

The National Archives says it is putting more servers online to handle the crush.  At one point, the Archives said, its computers were receiving 100,000 hits per second.

Hey, you've waited 72 years to see these records, so what's another day or two.

Earlier:

A time capsule from 1940 was opened on Monday at 9 a.m. ET, and we invite readers to share what they find. If you use the new records to find information about the loved or lost in your family, please post a note in the comments below or on our Open Channel page on Facebook.

U.S. Census records for individuals from April 1, 1940, protected until now by a 72-year privacy law, are now public for the first time, revealing details about millions of Americans from that day, as the country lingered in a Great Depression, still a year away from entry into war in Europe and the Pacific.

"I'm so excited!" Gary Robert Del Carlo of Martinsburg, W.Va., posted on Facebook. "Maybe for the first time ever, I'll be able to find out something about my father. All I have is my birth certificate with his name, date of birth, state born in, and that he was in the Army stationed in Washington State. His military records burned up in St. Louis in a fire in 1973. They would have told me a lot. Wrote for his birth certificate, and there was no records of his birth. I have done nothing but hit brick walls every which way I turn. I'm praying I find something useful tomorrow, anything."

NPR describes the release as the "Super Bowl for Genealogists." Librarians around the country are ready to provide assistance. At the Family History Library in Salt Lake City, the staff will be serving cake and providing help.

 

When the 120,000 census takers counted 132,164,569 people living in the country on that day, the information collected included the address, whether the house was owned or rented, value of the home or monthly rent, is it considered a farm, names of adults and children, familiy relationships, sex, race, age, place of birth, citizenship, residence five years earlier, education. And for a small subset of people, about 5 percent, they were asked about place of birth of mother and father, language spoken in the home as a child, veteran status, wars served in, Social Security status, occupation, employment status, occupation, number of weeks worked in 1939, income and, for women, whether they had been married more than once, age at first marriage, and number of children ever born.

There is a catch. As the records go online, they can't be searched by name. For a city it's helpful to know an exact address, but often you can work with a neighborhood (near the corner of Canal and Varrick streets in New York City). Your public library may have old city directories or telephone directories from that period, allowing you to look up people by name to find an address. For a rural area, you need to know at least the county and the name of the town or township.

Genealogists, librarians and volunteers will begin the work of indexing the records, which eventually will allow searches by name. Two sites, the commercial Ancestry.com and the Mormon Church's FamilySearch.org, have announced plans to provide indexes to their customers as quickly as possible, with some images going online on Monday. FamilySearch and Ancestry.com started putting images from the Census files online early on Monday, but for now without a name index. 

For now, you must know at least an approximate address to get started. You use that address to find an "enumeration district," which in a big city might be only a few blocks, and would be a larger area in a small town.

Another approach, for those interested in a specific place, is to look at all the records for your block or street. If your area was settled in 1940, who lived there then, and what were their lives like?

Your goal: With that district number, you can look on the Census website at the online copy of the form filled out by the census taker in 1940. In 70 years, it has gone from paper to microfilm to computer.

Here are resources to help you with the search (links open in a new window), though as with most things in life, the key is: Ask a librarian.

• Most important page No. 1: Step-by-step help from private researchers with free aids to help you find the enumeration district map for a particular address
• Most important page No. 2: A Census explainer on starting your search.
• The home of the 1940 Census
• A Census page with general information on the 1940 release
• A copy of the 1940 Census form (PDF file) that you can fill in when you find information
• Census aids to finding information
• Ancestry.com, a commercial service for genealogists
• FamilySearch.org from the Church of Jesus Christ of Latter-day Saints
• Tell us what you find: Post your story on Open Channel's Facebook page

 

In the field of climate science, when someone — especially skeptics — did something ethically questionable or misrepresented facts, scientist Peter Gleick was usually among the first and loudest to cry foul. He chaired a prominent scientific society's ethics committee. He created an award for what he considered lies about global warming.

Now Gleick admits that he posed as a board member to get and then distribute to the media sensitive documents from a conservative think tank that is a leader in questioning mainstream climate change science.

And ethicists are criticizing the man who took others to task for what they say was stepping way over the ethical line. The think tank, the Chicago-based Heartland Institute, said it is considering legal action against him.

---

Gleick, who won a MacArthur genius award and is co-founder of the Pacific Institute for Studies in Development, Environment and Security, was chairman of the American Geophysical Union's ethics committee. He also had a column at Forbes.com where he criticized climate skeptics and trumpeted the resignation of a scientific journal editor who published a disputed study. He admitted Monday night that he solicited and leaked the Heartland documents, writing in a blog post on The Huffington Post.

Gleick resigned from the chairmanship of the ethics panel last week.

"What a mess," said Mark Frankel, head of scientific responsibility for the American Association for the Advancement of Science, the world's leading scientific society, which also had Gleick as a panel member on some committees. "It's compounded by the fact that he was chairman of the ethics committee of a professional society. ... It's an ethical morass that he finds himself in."

And Gleick's actions cast unwarranted doubt on the work of other scientists, Frankel said.

Last week, someone identifying himself as "Heartland insider" sent 15 media members and others six documents, purportedly from Heartland. They included a fundraising document, a budget and a two-page "climate strategy." They showed the think tank receiving millions of dollars — more than $14 million over six years from one anonymous man — in big contributions with plans to teach school children to question mainstream climate science. It also showed funding of scientists who are climate-change skeptics.

Heartland said the two-page strategy document was a fake and the others were stolen. The Associated Press, which received the documents along with other news organizations, was able to verify the accuracy of several of the most sensational parts with the individuals named. The documents caused a stir, mirroring the hacking of climate scientists' emails two years earlier from a British research center.

"My judgment was blinded by my frustration with the ongoing efforts — often anonymous well-funded and coordinated — to attack climate science and scientists," Gleick wrote. "Nevertheless, I deeply regret my own actions in this case."

Not good enough, Heartland president Joseph Bast said in a press release: "It has caused major and permanent damage to the reputations of The Heartland Institute and many of the scientists, policy experts and organizations we work with."

The issue is about deception and there are only a few things that could possibly warrant that — and embarrassing Heartland isn't one of them, said Dani Elliott, who teaches ethics at the University of South Florida.

The geophysical union, a scientific society, said in a statement that Gleick's actions are "inconsistent with our organization's values."

Copyright 2011 The Associated Press. All rights reserved. This material may not be published, broadcast, rewritten or redistributed.

Updated: 4:40 p.m. ET on Feb. 15: The Heartland Institute says the documents referred to below were obtained through "pretexting," in which a person posing as a board member sent an e-mail asking a staffer to "resend" documents from board meetings. The Institute says one of the documents, a "climate strategy" memo, "is a total fake," and the institute says it has not had a chance to reach its president, who is traveling, to determine whether any of the other documents were altered. See the full statement from Heartland below. The group later said that the president had returned, that one document is definitely faked, and that it would not comment on the rest.

Internal documents have been leaked from the Heartland Institute, a Chicago nonprofit think tank, showing its funding of leading skeptics of global warming and a plan to teach climate change skepticism in schools. An anonymous person leaked the documents to several publications and activists supporting the science of climate change. 

"The heart of the climate denial machine relies on huge corporate and foundation funding from U.S. businesses, including Microsoft, Koch Industries, Altria (parent company of Philip Morris) RJR Tobacco and more," reports the DeSmogBlog, which published the documents on Tuesday. The blog opposes what it calls the "climate denial machine." (Disclosure: msnbc.com is a joint venture of Microsoft and NBC Universal.)

The first batch of documents is here on the DeSmogBlog, and a second batch dealing with fundraising.

---

The documents show a plan to develop a curriculum for teaching about climate change in K-12 schools:

Many people lament the absence of educational material suitable for K-12 students on global warming that isn’t alarmist or overtly political. Heartland has tried to make material available to teachers, but has had only limited success. Principals and teachers are heavily biased toward the alarmist perspective. Moreover, material for classroom use must be carefully written to meet curriculum guidelines, and the amount of time teachers have for supplemental material is steadily shrinking due to the spread of standardized tests in K-12 education.

Dr. David Wojick has presented Heartland a proposal to produce a global warming curriculum or K-12 schools that appears to have great potential for success. Dr. Wojick is a consultant with the Office of Scientific and Technical Information at the U.S. Department of Energy in the area of information and communication science. He has a Ph.D. in the philosophy of science and mathematical logic from the University of Pittsburgh and a B.S. in civil engineering from Carnegie Tech. He has been on the faculty of Carnegie Mellon and the staffs of the U.S. Office of Naval Research and the Naval Research Lab.

Dr. Wojick has conducted extensive research on environmental and science education for the Department of Energy. In the course of this research, he has identified what subjects and concepts teachers must teach, and in what order (year by year), in order to harmonize with national test requirements. He has contacts at virtually all the national organizations involved in producing, certifying, and promoting science curricula.

Dr. Wojick proposes to begin work on “modules” for grades 10-12 on climate change (“whether humans are changing the climate is a major scientific controversy”), climate models (“models are used to explore various hypotheses about how climate works. Their reliability is controversial”), and air pollution (“whether CO2 is a pollutant is controversial. It is the global food supply and natural emissions are 20 times higher than human emissions”).

Wojick would produce modules for Grades 7-9 on environmental impact (“environmental impact is often difficult to determine. For example there is a major controversy over whether or not humans are changing the weather”), for Grade 6 on water resources and weather systems, and so on.

We tentatively plan to pay Dr. Wojick $5,000 per module, about $25,000 a quarter, starting in the second quarter of 2012, for this work. The Anonymous Donor has pledged the first $100,000 for this project, and we will circulate a proposal to match and then expand upon that investment.

Here's a copy of the group's fundraising plan, with a list of donors.

The documents also show funding of leading voices among the opponents of the idea of global warming: "At the moment, this funding goes primarily to Craig Idso ($11,600 per month), Fred Singer ($5,000 per month, plus expenses), Robert Carter ($1,667 per month), and a number of other individuals, but we will consider expanding it, if funding can be found."

About its funders, the group refers to a single anonymous donor: "Our climate work is attractive to funders, especially our key Anonymous Donor (whose contribution dropped from $1,664,150 in 2010 to $979,000 in 2011 - about 20% of our total 2011 revenue). He has promised an increase in 2012…"

Other donors are named: "We will also pursue additional support from the Charles G. Koch Foundation. They returned as a Heartland donor in 2011 with a contribution of $200,000. We expect to push up their level of support in 2012 and gain access to their network of philanthropists, if our focus continues to align with their interests. Other contributions will be pursued for this work, especially from corporations whose interests are threatened by climate policies."

Statement from the Heartland Institute

Heartland Institute Responds to Stolen and Fake Documents

FEBRUARY 15, 2012 – The following statement from The Heartland Institute – a free-market think tank – may be used for attribution. For more information, contact Communications Director Jim Lakely at jlakely@heartland.org and 312/377-4000.

Yesterday afternoon, two advocacy groups posted online several documents they claimed were The Heartland Institute’s 2012 budget, fundraising, and strategy plans. Some of these documents were stolen from Heartland, at least one is a fake, and some may have been altered.

The stolen documents appear to have been written by Heartland’s president for a board meeting that took place on January 17. He was traveling at the time this story broke yesterday afternoon and still has not had the opportunity to read them all to see if they were altered. Therefore, the authenticity of those documents has not been confirmed.

Since then, the documents have been widely reposted on the Internet, again with no effort to confirm their authenticity.

One document, titled “Confidential Memo: 2012 Heartland Climate Strategy,” is a total fake apparently intended to defame and discredit The Heartland Institute. It was not written by anyone associated with The Heartland Institute. It does not express Heartland’s goals, plans, or tactics. It contains several obvious and gross misstatements of fact.

We respectfully ask all activists, bloggers, and other journalists to immediately remove all of these documents and any quotations taken from them, especially the fake “climate strategy” memo and any quotations from the same, from their blogs, Web sites, and publications, and to publish retractions.

The individuals who have commented so far on these documents did not wait for Heartland to confirm or deny the authenticity of the documents. We believe their actions constitute civil and possibly criminal offenses for which we plan to pursue charges and collect payment for damages, including damages to our reputation. We ask them in particular to immediately remove these documents and all statements about them from the blogs, Web sites, and publications, and to publish retractions.

How did this happen? The stolen documents were obtained by an unknown person who fraudulently assumed the identity of a Heartland board member and persuaded a staff member here to “re-send” board materials to a new email address. Identity theft and computer fraud are criminal offenses subject to imprisonment. We intend to find this person and see him or her put in prison for these crimes.

Apologies: The Heartland Institute apologizes to the donors whose identities were revealed by this theft. We promise anonymity to many of our donors, and we realize that the major reason these documents were stolen and faked was to make it more difficult for donors to support our work. We also apologize to Heartland staff, directors, and our allies in the fight to bring sound science to the global warming debate, who have had their privacy violated and their integrity impugned.

Lessons: Disagreement over the causes, consequences, and best policy responses to climate change runs deep. We understand that.

But honest disagreement should never be used to justify the criminal acts and fraud that occurred in the past 24 hours. As a matter of common decency and journalistic ethics, we ask everyone in the climate change debate to sit back and think about what just happened.

Those persons who posted these documents and wrote about them before we had a chance to comment on their authenticity should be ashamed of their deeds, and their bad behavior should be taken into account when judging their credibility now and in the future.

---

The document that Heartland says is a fake is this one titled "2012 Heartland Climate Strategy." The spokesman, Lakely, said it was defamatory to suggest that Heartland did not want science to be taught in schools, or that it would try to keep opposing views out of the press, or would think that it could.

The DeSmogBlog says about the "faked document":

The DeSmogBlog has reviewed that Strategy document and compared its content to other material we have in hand. It addresses five elements:

The Increased Climate Project Fundraising material is reproduced in and confirmed by Heartland's own budget.

The "Global Warming Curriculum for K-12 Classrooms" is also a Heartland budget item and has been confirmed independently by the author, Dr. David Wojick.

The Funding for Parallel Organizations; Funding for Selected Individuals Outside Heartland are both reproduced and confirmed in the Heartland budget. And Anthony Watts has confirmed independently the payments in Expanded Climate Communications.

The DeSmogBlog has received no direct communications from the Heartland Institute identifying any misstatement of fact in the "Climate Strategy" document and is therefore leaving the material available to those who may judge their content and veracity based on these and other sources.