The European Parliament's research department has found that four out of five member states surveyed carry out wide-scale telecommunications surveillance.
In a report released on Friday the department revealed that the U.K., France, Germany and Sweden all engaged in bulk collection of data. The Netherlands, which was also examined, has not done so, so far, but is engaged in setting up an agency for that purpose.
The report notes that although surveillance has been carried out for decades, there is no room for complacency because the amount of data currently available is so large. It says the current surveillance programs "go largely beyond what was called before targeted surveillance or a non-centralised and heterogeneous assemblage of forms of surveillance."
The U.K. leads the European surveillance field and is the only country to come close to the scale of the U.S. National Security Agency (NSA), the report says.
In the U.K., the Government Communication Headquarters (GCHQ) receives approximately £1 billion (US$1.6 billion) annually and has a staff of 6,000.
"It appears unlikely that the programmes of EU member states such as Sweden, France and Germany come close to the sheer magnitude of the operations launched by GCHQ and the NSA," says the report.
Reports allege that GCHQ has placed data interceptors on approximately 200 U.K. fiber-optic cables that transmit Internet data and that by 2012 the agency was able to process data from at least 46 fiber-optic cables at any one time. This gives the agency the possibility to intercept more than 21 petabytes of data a day. This is estimated to have contributed to a 7,000 percent increase in the amount of personal data available to GCHQ from Internet and mobile traffic in the past five years.
In order to deal with this vast amount of data, GCHQ uses a system of so-called "Massive Volume Reduction," removing 30 percent of less intelligence-relevant data such as peer-to-peer downloads. The remaining data is combed using some of up to 40,000 "selectors" such as keywords, email addresses or phone numbers of targeted individuals by about 300 GCHQ and 250 NSA staff working together.
Content such as recordings of phone calls, content of email messages and entries on Facebook is kept for up to three days while metacontent such as time, date, creator and location of content is stored for up to 30 days.
In France, the DGSE (Direction générale de la sécurité extérieure) is responsible for surveillance. In 2010, Bernard Barbier, a technical director at the DGSE, said that France ranked fifth in the world in metadata collection after the U.S., the U.K., Israel and China, and runs the second most important intelligence data collection and processing center in Europe after the U.K.
Data is intercepted and collected by approximately 20 interception sites, located on national and overseas territories and comprised of satellite stations and interception of fibre-optic submarine cables.
In Sweden, the National Defence Radio Establishment (FRA) is alleged to have been running "upstreaming" operations (tapping directly into the communications infrastructure as a means to intercept data) for the collection of private data -- collecting both the content of messages as well as metadata of communications crossing Swedish borders through fibre-optic cables from the Baltic Sea. The metadata is retained in bulk and stored in a database known as "Titan" for a period of 18 months.
In Germany, large-scale surveillance activities are predominantly carried out by the Bundesnachrichtendienst (BND), which has a staff of 6,500 and last year had a budget of €504.8 million (US$694 million). Two other organizations also believed to be running mass surveillance operations or processing related data are Militärischen Abschirmdienst (MAD) and the Bundesamt für Verfassungsschutz (BfV).
The BfV employs 2,757 people and had a budget of €210 million in 2012. The three intelligence agencies together search up to 20 percent of communications having a foreign element for specific purposes such as the fight against terrorism or the protection of the Constitution.
The report notes that there are currently no publicly disclosed programs of mass cybersurveillance in the Netherlands. However, the Joint Sigint Cyber Unit (JSCU) is due to be up and running next year. It is expected to centralize cybersurveillance in the Netherlands and will have a staff of 350. Its annual budget is unknown, but it will cost €17 million to set up.
The official objective of the program is the infiltration of computers and networks to acquire data for early-warning intelligence products; the composition of a cyberthreat picture; enhancing the intelligence; and conducting counterintelligence activities.
According to the Parliament report, there are strong suggestions to indicate that several if not all of these member states are exchanging intercepted data with foreign intelligence services, namely the NSA.
"At a very pragmatic level, large-scale surveillance appears to have strong limitations and is certainly not key in crime prevention. Such surveillance creates the tendency to collect data extensively and retain them over a long period of time in order to establish series of trends that facilitates big data correlations and hierarchies," the report said.
However, the report is concerned that the distinction between targeted surveillance for criminal investigations purposes and large-scale surveillance with unclear objectives is increasingly blurred, and recommends further investigation.
Join the CIO New Zealand group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.