Voices on Values survey: brief overview of methodology

junior ferreira 735237 creds

The survey involves six countries - Hungary, Poland, Greece, Italy, France and Germany.

Its questionnaire has two main goals: first, to better understand attitudes to open and closed societies, and second to examine how people respond when asked to evaluate the comparative open and closed society merits.



 What makes a good society

In the first part of the survey respondents were presented with 14 statements – half associated with open society views, and half with closed society ones. They were asked to rate each statement. For example:

  1. Should everyone be allowed to practise their religion freely?
  2. Should government ensure that media reporting always reflects a positive image of a country?


Trade-offs within a good society

For each of the first part’s seven open society statements, its second part showed two statements and asked respondents to evaluate their relative importance for a good society. They were asked to choose one, the other or both.

For example, in the statement that everyone should be able to express their opinion freely, the alternatives were:

  1. That Christian values not be offended
  2. That ethnic and national minorities not be offended.

 The survey also asked views on about immigration, civil society and political affiliation, with a few country-specific questions.


The development team

The research institute d|part developed the survey in close cooperation with OSEPI and the six country partners. The final survey was written and edited in English, before being translated into the six relevant languages.

Fieldwork was done by Lightspeed Germany, and the survey was administered via an online panel representing the six countries. Each country had quotas for ages, gender, geography, education and income levels. A soft launch pilot was carried out with 50 respondents, and the full launch includes more than 1,000 respondents in each country. The survey took place between February 12 and March 5, 2018.


The extended explanation

This note briefly summarises the background of the survey that was carried out to collect data on open society attitudes across the six countries in the project (Hungary, Poland, Greece, Italy, France and Germany). The questionnaire was developed to address two main areas of concerns: first, to evaluate the composition of open and closed society attitudes conceptually and second, to examine how individuals respond when asked to decide whether a particular open society attribute was more, less or equally important than an attribute associated with closed societies.

Open society constructions

In the first part of the survey respondents were presented with 14 items (in random order) and asked to indicate how important they thought the respective item was for a good society. Half of the items (7) were attributes defined as open society characteristics and half (7) were attributes more closely associated with closed societies.

The data collected in this first section allows us to apply dimension reduction techniques to examine how the items relate to each other. The items were:

Based on the answer options we computed two scores, one for the rating of open and one for closed society attributes. To calculate the score, the order of the answer scales was reversed (so higher values indicated higher levels of rating an item as essential). Then all seven respective item scores were added up, resulting in a score between 7 and 28. We then standardised the scores between 0 and 1. So both scores respectively measure how essential respondents rated the open and closed society attribute items respectively. A score of 0 means respondents said “not at all essential” to all seven items, a score of 1 means they said “absolutely essential” to all seven items.

Trade-off experiments

In the second part of the survey respondents were presented with 14 direct comparisons between two items and asked to evaluate their relative importance. For each of the seven open society attributes from the first part of the survey, respondents were presented with two alternative items and asked to make a decision in each of those comparisons. The order in which the comparisons were presented was randomised. The full set of comparisons was as follows:

Table 2

 Other questions

In addition to the instruments presented above, we also included a number of socio-economic questions and correlate questions about attitudinal domains, such as attitudes towards immigration, civil society and their political affiliations across all countries. Additionally, a few country-specific questions were added in each country to support the analyses within each country-specific context.

Survey development

The survey was developed by dpart’s core team for the Voices on Values project and in close cooperation with all five country partners. After initial scoping about issues and instruments, a draft questionnaire was developed that was then discussed in a workshop with representatives of all country partners, d|part and OSEPI. Based on that workshop a second draft questionnaire was developed that went through further iterations with feedback from all partners. The final survey (in an English master version) was then translated by professional translators into the six country languages. Country partners then checked the translation, in addition to checks carried out by the core team. Feedback was then given to the translators to further improve the translations. Country partners formulated draft questions specific to their own contexts that were reviewed and edited by the core team before going through a final round of checks together with the master questionnaire.

 Fieldwork and data

The fieldwork was carried out by Lightspeed Germany in close cooperation with the core team. The survey was administered through an online panel in all countries. Programming of the survey was pre-tested by several people in each country to check for user experience, correct routing and the implementation of translations. Quotas for age, gender, geography, education and income and several cross quotas were applied, to achieve good representation in the samples. Quotas were only relaxed at later stages in the fieldwork and in case they could not be filled adequately otherwise. Before commencing with the fieldwork, a soft launch pilot was carried out with 50 respondents in each country to test the survey instruments and check initial distributions and participation. Subsequently the full launch took place with over 1000 respondents recruited in each country. The survey was carried out between 12 February 2018 and 5 March 2018. Where achieved sample distributions deviated from actual population distributions, weights were calculated and applied to account for those deviations.

The open society: elusive but essential

By Daniel Devine 

samuel zeller 358865 credsThe rise of populist parties across Europe and disaffection with mainstream politics are said to endanger the open society. But what is an open society? A new project by OSEPI and d|part, supported by five country partners, tries to find out what open society means to the citizens of six European countries, and what trade-offs they are willing to make to achieve it.

Whether the open versus closed divide is defined as cosmopolitans vs non-cosmopolitans, the nationalists vs globalists, or somewheres vs anywheres, understanding its causes and manifestations is key to grasping European politics today. The Voices on Values project shines its spotlight on areas where the open society is most at risk, and attempts to do so by finding where citizens and elites place the limits between open and closed societies.

The term open society was first coined by Henri Bergson in 1932, and popularised six years later by Karl Popper in his book The Open Society and its Enemies. Both agreed that closed societies value security, self-preservation, conformism, nativism and tribalism, The two didn’t see eye to eye on what they meant by open society. Bergson defined it as one “which will, in principle, embrace all humanity” whereas Popper saw it as meaning “faith in man, in equalitarian justice, and in human reason”. They did, however, agree that its institutions were those of liberal democracies.

An open society has become an elusive concept in today’s Europe: there is little consensus about its core values, or on what separates open and closed societies. An open society can be defended with apparently contradictory arguments. In a discussion about plans to ban the burka, Germany’s Interior Minister Thomas de Maizière said, “We are an open society […] we’re not burka”, arguing that “to ‘show one’s face’ is the expression of democratic coexistence”. Does this mean that de Maizière sees conformity as an open society value, when others might argue that freedom of expression is much more important.

It is not just where lines are drawn but also for what reasons. When british Lib-Dem politician Nick Clegg evokes social mobility as an open society value, he is stretching the expression’s commonly assumed meaning. But the concept’s elasticity is one of its virtues, as it can be reformulated to meet changing political conditions.

Daniel Devine is a PhD student at the University of Southampton and research consultant to OSEPI.


The views and opinions expressed in this article are those of the author, Daniel Devine.



About Voices on Values

A joint project of the Open Society European Policy Institute (OSEPI) and d|part


kelly sikkema 411622 creds

A variety of societal changes challenge our views of open society principles. They include the dissatisfaction with conventional politics expressed by populist parties; increased concerns over security and immigration; nationalism and undemocratic policy changes promoted by the media and endorsed by the public at large. Civil society groups and policy-makers are mobilising to defend an open society, yet many disagree about what it actually means.

Aim of the Voices on Values project

To help civil society organisations along with national and EU-level political actors, as well as the wider public, respond to these profound political disruptions, the Voices on Values project seeks to understand what drives the changing discourse about open societies and the mechanisms that can be used to strengthen them. It puts the spotlight on six countries: Germany and France in North-western Europe, Hungary and Poland in Eastern Europe, and Italy and Greece in the Mediterranean.

In each of these countries, researchers have been exploring the relationship between four areas: policy; communication and representation; the response of civil society; and public attitudes. The project focuses on the complex and dynamic relationship between these areas, such as how policy changes affect political communication, how this in turn affects public opinion, and how civil society can respond to changing law and attitudes.


This research is based on interviews with decision-makers and civil society leaders and public surveys conducted across all six countries. In these we asked people across Europe about the importance of values associated with open societies, such as freedom of expression, press freedom and the freedom of religion, and through an experimental survey design tested to what extent people's evaluations are robust. 

Getting heard

Working with a network of think-tankers and researchers in six countries, the Voices on Values project produces regular analysis, including opinion pieces and shorter data-based articles on themes like young people and civil rights that are the basis of in-depth reports from each country. Additionally, in February 2019 six individual country reports, as well as three reports on the project's more general findings will be published.