Rethinking Cultural Evaluation

By Dr. Cristina Ortega Neure

Director, Institute of Leisure Studies. University of Deusto

and Dr. Fernando Bayón

Senior researcher. University of Deusto

/Proceedings

Rethinking Cultural Evaluation: Going Beyond GDP.

Advance Seminar: Rethinking Cultural Evaluation: Going Beyond GDP

Organizers: European network on Cultural Management and Cultural Policy Education (ENCATC)

Scientific Coordinator: Prof. Cristina Ortega Nuere, Chair of the ENCATC Thematic Area “Monitors of culture”

Venue: Paris, France

Main focus of the discussion: A discussion follow on rethinking on how to measure the spill‐over effects of cultural and creative industries and how to evaluate cultural approaches. This event offered participants the possibility to understand the most recent advancements in the debate on culture as an indicator of well‐being and development.

Introduction

On 23 of October, 50 researchers, academics, cultural operators and practitioners, cultural managers and representatives from European and international institutions met in Paris to discuss about the most recent advancements in cultural evaluation. All in all more than 11 countries were represented with participants coming from as far as Morocco and Australia. Among the audience were representatives of the European Commission, UNESCO, OECD, the European Cultural Foundation, the French Ministry of Culture, the British Council, the Louvre Museum, as well as directors of prominent cultural networks and art organizations.

Organised by ENCATC in partnership with the University of Deusto, BizkaiLab and the Diputación Foral de Bizkaia, this event was hosted by Vivendi Universal. After the seminars “Rethinking Cultural Evaluation: Going Beyond GDP” (Antwerp, 2013) and “Place of culture in sustainable development: going beyond the GDP indicators” (New York City, 2013), the debate on cultural indicators has continued in 2014 with an Advanced Seminar organized in Paris. This new debate in Paris was a unique platform for the major players on this topic to share their research and results from their respective policies and projects on evaluation in the arts and culture. Participants gained insight into the most recent advancements in the debate on culture as an indicator of well‐being and development. A discussion followed on rethinking how to measure the spill‐over effects of cultural and creative industries and how to evaluate cultural approaches.

This seminar was an initiative of ENCATC and an outcome of the ENCATC Thematic Area “Monitors of Culture” led by professor Cristina Ortega Nuere, Director of the Institute of Leisure Studies at the University of Deusto in Spain. The ENCATC Thematic Areas were set up in 2007 to offer as a space for specialized knowledge in specific fields of Culture and External Relations – Diplomacy, Heritage, Museums, Performing Arts, Arts and Wellbeing, Creative Industries, Cultural Data and Evaluation, and Urban Management.

The hub fostered by the ENCATC Thematic Area “Monitors of Culture” intends to be a space where renowned experts, academics and stakeholders in creative and innovative policies meet to explore new dimensions of cultural evaluation as a political challenge across Europe. In the specific case of the advance seminar “Rethinking Cultural Evaluation: Going Beyond GDP” international experts from institutions such as UNESCO, OECD, Joint Research Centre; universities such as the University of Deusto and Erasmus University Rotterdam; enterprises such as VIVENDI; consultancies as KEA; research networks like the Cultural Development Network; or other organizations such as the European Centre of Creative Economy and Forum d´Avignon presented their most recent contributions to the state of art of cultural evaluation with an international approach and shared their experiences and good practice at local and trans-regional levels.

The advanced seminar started with some introductory words about the present challenges to develop further indicators about the intangible and immaterial effects of CCIs as well as new methodologies taking advantage of the potential of TIC and artists’ participation by Cristina Ortega Nuere, Director of the Institute of Leisure Studies of the University of Deusto. The seminar continued with a first panel about open frameworks where experts such as Fernando Bayón, researcher of the official Leisure and Human Development research team of the University of Deusto; Melika Medici Caucino, Programme Specialist, Division of Creativity, UNESCO; María Iglesias Portela, Head of Research and Analysis-KEA European Affairs; Lorena Sánchez, Project Coordinator of Better Life Initiative, OECD, contributed to a more complex perspective about the measurement of the impacts on culture, with special attention to the diversity of its social dimensions and the richness of its spillover effects. Afterwards, a second panel made up of researchers and professionals such as Kim Dunphy, Research Programme Manager at the Cultural Development Network of Australia; Claudine de With, Researcher at Erasmus University Rotterdam of Netherlands and Pascale Thurmerelle, Vice President CSR at Vivendi discussed about innovative, participative and co-creative methodologies inspired in community engagement. The seminar ended with an open discussion about the new territories of culture, where experts such as Dorota Weziak-Bialowolska, Coordinator of the Cultural and Creative industries Activity of the European Commission-Joint Research Centre; Olivier Le Gualy, Editorial Manager at the Forum de´Avignon; and Bernd Fesel, Senior Advisor at the European Centre for Creative Economy (e.c.c.e.) introduced the newest debates about emerging territories of creativity in the global and digital world, with special attention to the role that CCI’s and ICTs undertake.

Background

The UN General Assembly in its Resolution 65/309 entitled “Happiness: Towards a Holistic Approach to Development” was conscious that the pursuit of happiness was a fundamental human goal and recognized that the indicator of gross domestic product (GDP) was not designed to and did not adequately reflect people’s happiness and well-being. Consequently, the Assembly invited Member States to draft additional measures that could better capture the importance of the pursuit of happiness and well-being in development, with a view to guiding their public policies. Main Member State initiatives to develop new indicators were brought together in the aforementioned Resolution. Another initiative that has led to significant progress in this regard is the Stiglitz-Sen-Fitoussi Report on rethinking GDP produced by the Commission on the Measurement of Economic Performance and Social Progress, where the shift in emphasis from measuring economic production to measuring people’s well-being is remarked upon.

“Cultural and creative industries’ (CCIs) can stimulate other processes that go beyond economic development and are likely to generate innovation in sectors other than culture.”

Cultural and creative industries’ (CCIs) economic potential is widely recognized: in the EU they account for 3.3% of the GDP and employ 6.7 million people, representing 3% of total employment. Besides their direct contribution to jobs and growth in regions and cities, CCIs can stimulate other processes that go beyond economic development and are likely to generate innovation in sectors other than culture. Spillover effects (SOEs) deliver excellent results in a wide range of fields including innovative SMEs, urban regeneration, climate change, quality of education, creativity, social innovation, social cohesion, skilled employees, gender equality, well-being, resilience, human development, etc. There is an increasing awareness of the great diversity of spillover effects that CCIs can have on the wider economy, both on society and individuals.

Measurement of spillover effects has been focused on the economic field. Therefore, the indicators developed to enable us to evaluate CCIs’ impact have been mainly economic, specially oriented to the study of GDP impact and employability. Social impacts of CCIs have also been taken into consideration by a more recent generation of indicators. In this sense, developed indicators go beyond GDP – post-GDP indicators- in order to take into consideration effects such as social inclusion or gentrification. From the methodological point of view applied to develop these indicators, economic and more recent indicators -such as the social ones- are both created from methodologies that do not take into consideration the voice, role and needs of the stakeholders involved, which is necessary indeed.

There is little research that has overcome the economic and social perspective including, for instance, impact on human development factors – psychological well-being, active aging, innovation and social innovation. One of the reasons for the lack of these indicators is probably their indirect and intangible nature or quality. It is indeed a challenge for the social sciences to develop indicators to measure immaterial and indirect effects.

Framework and methods

Practices performed along the advanced seminar held in Paris were setting up for two panels and a roundtable discussion. Panels were called “open frameworks” and “new methodologies” and are summarized in the following paragraphs. By means of these panels as well as the round table the seminar aimed to gather and coordinate the two levels in which the current of cultural evaluation is pivoted: the conceptual framework and applied methodologies. None of them can be understood separately. The first panel tried to answer to the questions “what should we understand by cultural impact?” and the second one “what methodologies allow us to test the new indicators of these impacts?”

Culture for Development Indicators, UNESCO

Melika Medicci, Programme Specialist, Division of Creativity, UNESCO, presented the work carried out by UNESCO, specifically from the specialist division of creativity. That institution has been researching about the relation between culture and development for a long time ago and, at the present moment, their work is focused on the pursuit of culture for development DNA, stepping over in the delimitation of a culture methodology for development. Demonstrating with facts and figures are the contribution of culture in development processes and assessing the environment for enhancing and sustaining cultural assets, resources, and processes for development are the main objectives that this research seeks. Along the presentation were exposed twenty-two indicators, which are approaching to measure the seven dimensions, consider for the method to achieve main aims in the field. Concerning the dimensions, there are two kinds of indicators (benchmarks and descriptive) that move across vital dimensions: economy, education, governance, social participation, gender equality, communication and heritage. The research has been tested in 11/12 countries and the collaboration of 150 partners. For further information visit: http://en.unesco.org/creativity/cdis/

Evaluation of European Capitals of Culture, KEA

María Iglesias Portela, Head of Research and Analysis – KEA European Affairs presented some insights about the evaluation of European Capitals of Culture. The European Commission has carried out many evaluations on the European level on its successful ECoC initiative. Notable evaluations come from: 1994 (Lisbon) – 2004 (Lille) – 2006 (Patras). These evaluations have been complemented with evaluations done by the cities themselves that take into account the local priorities that aren’t represented in the EC evaluations. As a consequence, there is a lack of comparable data on ECoC since cities’ reports are made up of different targets, contexts, frameworks and methodologies, etc. However, evaluations will be even more important and are part of the new EU rules. ECoCs must integrate cultural policy into their applications and already have an evaluation strategy and indicators to measure the impact of their activities and actions. It will also be important to: a) to promote close collaborations with other sectors; and b) to increase citizen awareness and participation. For further information visit: www.keanet.eu

The Better Life Initiative Index, OECD

Lorena Sanchez, Project Coordinator of Better Life Initiative, presented OECD’s Index on Better Life. OECD’s focus has been on GDP and not on culture, although brining culture in has been a hot discussion topic. As culture is not yet a dimension, should it be added? Actually it is more of a question of when it will be added. We are now at the beginning stages of culture becoming a dimension. The initiative undertaken by the OECD in terms of this approximation towards a cultural dimension that can introduce new components based on quality of life, human development and well-being to be defined as “the better life initiative”. There are five contended features, which are related to: people, outcomes, averages and inequalities, objectives and subjective aspects and time aspects (distinguish between today and tomorrow and the generation of future sources of “good outcomes” that people will be able to achieve).This entire work starts answering a question: “How´s life?” in 2011, first edition, where it is provided a set of internationally comparable indicators to measure well being in 40 countries (OECD and BRIICS). At the second edition in 2013 it includes new components in the analysis concerning human costs on the finance crisis, well-being in the workplace and the importance of quality jobs, gender inequality in well-being and, finally, on sustaining well-being over time.
For further information visit: www.oecd.org/statistics/better-life-initiative.htm and
www.oecdbetterlifeindex.org

Holistic model of outcome evaluation for arts engagement, CDN Australia

Kim Dunphy, Research Programme Manager at the Cultural Development Network of Australia, introduced a holistic model of outcome evaluation for arts engagement developed by this independent non-profit organization held in Melbourne. For them it is essential to create links between local government, communities, artists and related agencies into the own process of/for culture, indeed, in their own words: “we advocate for the essential function of arts and cultural expression in the development of creative, healthy, engaged and sustainable communities. We support local government in their role of assisting and resourcing local communities to make and express their own culture”. According to the holistic model presented, it starts questioning some problems about outcomes: which perspectives, data and subjective direction and dimension. It represents a global circle divided in five sets or domains, as called, about ecological, cultural, civic, social and economic aspects. Within each, it is defining seven components bringing us to a complete and complex perspective for the real change and expected one. Community Indicators Victoria domains of local development have inspired it.
For further information visit: www.culturaldevelopment.net.au and www.communityindicators.net.au

Measuring the quality impact, Erasmus University Rotterdam

Claudine de With, Researcher at Erasmus University Rotterdam, presented a project focused on measuring the quality impact: “an evaluation framework that provides an assessment of merit (quality) of cultural policies and activities”, assume that this information help the cultural institutions and policymakers. The approach for a monitor of quality impact of cultural policies is given through four steps, in form of questions: What does the organization aim to achieve? What does the organization possess in order to achieve the values or ambitions? Who is on the receiving end of the organization’s activities? How to make Quality Impact insightful? The design of this approach in terms of the artistic and social value of Dutch arts centers showed during the seminar gives another way to tackle/address the measurement of quality of cultural policies. For further information visit: www.aemuse.nl/encatc-seminar-paris/ and www.aemuse.nl

Keys to a sustainable creativity impact model, Vivendi

“CSR priority areas on cultural dimension: promoting cultural diversity, protecting and empowering young people and knowledge sharing.”

Pascale Thumerelle, Vice President CSR at Vivendi, introduced the private sector perspective to the debate. The insight from Vivendi in its presentation is facing across a media and content business where it is added value through corporate social responsibility (CSR). For this company CSR is articulated in ten priority areas for action of which three of them are emphasized into a cultural dimension: promoting cultural diversity, protecting and empowering young people and knowledge sharing. These three strategic issues stemming from human rights and other institutional reports support “the responsibility of enterprises for their impact on society” and are putting them into practice through a project to integrate them since 2003. The Head of CSR explained the project along last years and how they obtain very higher results, economical and socially, in comparison with others companies in the sector; through specific media indicators. For further information visit: www.vivendi.com and http://www.vivendi.com/responsabilite-societale/

New horizons of empirical research of cultural impacts of JRC (EC)

“More tailor-made research is needed to explore the impact of the CCIs on other industries and on society.”

Dorota Weziak-Bialowolska’s from the European Commission, Joint Research Centre (JRC), introduced the work done in the Econometrics and Applied Statistics Unit Composite Indicators Research Group (JRC-COIN). She introduced the debate the contribution of Empirical research to out knowledge on the CCI. She pointed out the current difficulties to find data on CCIs. However, this situation will be improved with the Modernization and Social Statistics Unit that has been set up in the Eurostat to deal with cultural data; and the Expert group on cultural statistics (composed of National Statistical Offices and possibly Ministries of Culture) that will start working in April 2015. She remarked the fact that more tailor-made research is needed to explore the impact of the CCI on other industries and on society. She finished her presentation with some examples of composite indicators underlining the need to integrate science and technology with arts.

A critic consideration of evaluation practices, e.c.c.e.

Bernd Fesel, Senior Advisor at the European Centre for Creative Economy (e.c.c.e.) introduced some reflections about the social sustainability of societies. Qualitative research leads to idea production and it is also a response to participatory needs. But how can we motivate citizens to participate in urban change? How can we combine quantitative and qualitative research? It is fundamental the kind of vision we have of society that will dictate or influence the kind of research that is carried out. The European Commission has also cited in its Europe 2020 strategy the importance of social cohesion and a sustainable society. The Communication “Europe 2020 A strategy for smart, sustainable and inclusive growth” was surprising because it went beyond (or at least a little bit) the pure economic wording and ideology. Now perhaps is the time for researchers to take advantage of this. Culture is not about making an economically efficient society – it is about making a sustainable social society. For further information visit: www.e-c-c-e.de

Rethinking the experience of Forum d’Avignon

Olivier Le Guay, Editorial manager at the Forum d’Avignon from France introduced some issues for reflection. Is measurement of culture’s spillover a utopian idea? Forum d’ Avignon sought to take a look at spillover. Culture is a strong driver for the economy, but in order to take action we need figures since evaluations lead to decisions. Figures and research are also important for comparing the cultural sector with other sectors. This drives the point of the importance of creativity. It’s interesting to see how cultural indicators are completely spread all over the place. We need people who try to bring them all back. Together they can have a greater impact. Can measurement be dangerous for creativity? Public and private players need to work more together. What is our cultural footprint and what will this mean for future generations? As new funds become available this is the time to review where we are standing. And also to ask: What is our vision? It is crucial to bring in researchers, practitioners, policy makers, cultural players into these actions. If we invest in accessible culture it is possible to change the dimension of the local territory. But it is more than building a new museum – these decisions need to be strategic. A culture of change will go beyond the political term of the person who initiated it. This is why strategy is so important. For further information visit: www.forum-avignon.org

Current debates and future challenges

After a brief introduction, the Advanced Seminar started with a panel on open frameworks followed by a debate on methodologies and a final round table where discussion was focused on the following issues.

It is difficult to find existing indicators that are harmonised and comparable. During panel discussions several experiences about the process of developing indicators were shared. Besides the difficulties faced to measure culture, such as the diversity and broad scope, measuring spillover effects have to overcome further difficulties to make tangible the intangible and measure the indirect effects of CCIs such as subjective wellbeing. One of the first issues arose was the need to have a dialogue among the scientific community, policy makers, private sector and society. Special attention was given to the lack of participation of artists from the beginning of the process when designing indicators as well as the value of social narratives.

Relevance of testing and developing context based tailored made indicators

During the discussion some issues such as the relevance of testing and developing context based tailored indicators were remarked. Indicators developed by experts in a meeting room or as a result of desk research do not always work once you get into the field. For international organizations this is particularly difficult because they are working with so many regions worldwide that some of the indicators are not very meaningful for some of the countries, but for others they are. Countries are usually encouraged to design additional indicators – more tailored – that reflect their unique situations in order to complement the international frameworks. These indicators should be elaborated in a participatory way in order to better interpret data. This data is crucial for facilitating broader discussions with both cultural and non cultural players because the discussions, interpretations, priorities, etc. are different from country to country.

Trans-sectorial thinking and dialogue

Learning from or communicating with other territories, organisations. Which indicators and data resources are they using? What methodologies are they developing? Collaboration with other sectors was highlighted during the discussions. We look for indicators among indexes that have been developed from other sectors such as innovation or human development. However, cultural indicators can also been highly valuable for other sectors. Dialogue between sectors and more collaboration among public, private, civil society, regions and cities should be highly encouraged to reflect together on how to advance measurement. Sometimes bringing different people together who usually do not dialogue around the same table is more important than the research results.

From participation to co-creation

How can this be done? From the very beginning, involving stakeholders: artists, cultural managers, citizens, local organisations, etc. – all these actors need to be engaged and feel part of the discussion. This is a way to fine tune some of the indicators and data collection qualities. It is also a way to engage with new audiences and empower the population. The more you know about what and who you are evaluating the more you can identify, adapt and frame your evaluation.

There are some experiences like the OECD index where any citizen can contribute to the discussion. Right now citizens can do two things with the OECD index: they can remove an indicator completely from factoring into the index and/or participants can add a comment for additional dimensions or indictors that they think are important. Being aware that it is enough, when the extended survey is constructed this should be more upfront. Based on the comments received to date, many people mention culture, the weather/climate.

Regarding the nature of stakeholders involved in the development of international indicators, most of them are institutional: national and statistic offices, ministries of culture, professional associations, social affairs, gender, employment, etc. However, some professional organizations are not used to working with indicators. Including them in national workshops -when building frameworks and indicators- means a process of building capacities together -that is the case of CDIS. In other cases such as ECoC, cultural managers have been included in the process of evaluation. We need the sector and players to take ownership and we also need to better inform associations who aren’t used to using this data.

From big to small data

It is difficult to go more deeply at the national level with the surveys and data sources we have to evaluate culture’s spillover effects. It’s difficult to know who has answered. Have artists and creators responded? It is important that cities are involved because they can provide new insight where national evaluations cannot provide a better understanding. At the municipal level it is easier to work with cultural institutions, gather more detailed data and work with focus groups in order to really be able come up with new indicators.

Transforming intangible into the tangible

How can we look at qualities and if they are being achieved? The cultural sector is different from other sectors, among other issues because there are a lot of non-profit organisations which mean they have qualities that require other indicators. A value-based approach is required here, but it can be difficult to talk about values. Often it is easier for people to discuss not what values they have, but about what it is they want to achieve/goals. How do we translate these goals into values? Here language is also very important because not everyone uses the same definitions or has the same terminology to talk about values, objectives, aims, goals.

We need to find ways to turn qualitative data into quantitative. An example given: when a patient in a hospital is asked to rate their pain level on a scale of 1-10 this is a perfectly acceptable and considered a sound measurement of pain. Why can’t it be the same for people’s feelings about culture? What needs to be done is to develop a framework where data can be entered. And the other way around we need new social narratives that could give new political senses or meanings to the pure figures.

Panels ended with many other reflections and questions posed by participants that might need further reflection and be discussed in next debates and seminars: How can we better connect people, business, environment, research centres? How can technology play a role? This has a great potential to get more people to participate: How can we invent/create new methodologies and compare them? How can we teach/inform cultural managers to use these evaluation tools? How can we better report data? How can organisations publish their results? Open access? How to measure/encourage civil participation taking into consideration that there are three indicators for a cultural dimension that reflect values of society from the point of citizens?: a) Access to cultural products and services, (b) Contexts favourable for one to develop his or her own creativity and enjoying abilities to be creative, (c) Contribution of media to generate the development of society. Culture has been around for thousands of years. We ourselves can be barriers in the measurement of culture’s spillover effects. Is it a question about ideology? If we can figure out a way to measure quantum physics, we should be able to measure culture.

Most relevant conclusions

The debate concluded remarking the value of the following issues: awareness, rethinking and better understanding of CCI’s measurement.

• Awareness: Awareness about how CCIs can stimulate other processes and contribute to more than just economic growth should be increased. The value of CCIs to development beyond GDP -such as human and societal development- should be enhanced.

• Rethinking: Rethinking development measurement beyond GDP demands rethinking not only the frameworks and indicators but also the methodologies. At this stage traditional approaches no longer apply and need to be rethought. We should respond to the drastic changes and advancements in technology and take advantage of TIC to better measure spillover effects since society is rapidly changing and the potential of new technologies has been undermined. More research needs to be done on: a) common definition/understanding of spillover effects; and b) new measurements “beyond GDP”: Frameworks, indicators and methodologies.

• Further transfer of knowledge, understanding and empowerment: There is a lack of knowledge of the potential, value and existence of indicators among organizations, associations, entrepreneurs, etc. They should be better informed on how to use collected and analysed data – they may not be used to using these resources. The sector and actors should also be encouraged to take ownership and be more involved. Feedback from citizens, artists, and those who can use this data and analysis is requested as well. Furthermore, awareness should be increase so that research gets into the hands and is read by those who can benefit the most.

​Cited as: Ortega Nuere, C. and Bayón, F. (2014). “Rethinking Cultural Evaluation: Going Beyond GDP”. /encatcscholar, number 3, January, 2015. ENCATC. Retrieved from: http:www.encatc.org/encatcscholar. Accessed January 31, 2015.