Variable Selection and Data Quality Challenges in Impact Assessments
DOI:
https://doi.org/10.18662/po/12.3Sup1/348Keywords:
impact evaluation, counterfactual impact evaluation, variables, data quality, big data, blockchainAbstract
The research is focused on the role of two related key concepts, namely variables and data, in the impact evaluations of public projects. A difficult task of the evaluators and researchers is to select the appropriate variables to ensure the best model of reality and satisfy the evaluation methods' needs. Therefore, the paper aims to look at the current knowledge and discuss how variables and data could be best used to connect the evaluation models, the particularities of the intervention with the potential of the advanced quantitative assessment methods.
The results emphasise that evaluations operate with data with different levels of granularity, as required by the intervention logic. Structuring data in clusters and categories, performing evaluability assessments are useful in assessing data quality and limitations and improving them. In line with the existing literature, we demonstrate that data accessibility is a key constraint and imposes adjustment of the desired evaluation model to a feasible one.
While Big Data and Open Data systems significantly improved data quality in evaluations in recent years, blockchain, as a ledger technology with default features related to decentralisation and security, is expected to bring large benefits to evaluation. For evaluators and policymakers, blockchain potential is an area of further research looking for additional advantages that could enhance the use of quantitative methods.
References
Befani, B., Barnett, C., & Stern, E. (2014). Introduction–Rethinking impact evaluation for development. IDS Bulletin, 45(6), 1-5. https://doi.org/10.1111/1759-5436.12108
Berman, E., & Wang, X. (2012). Essential statistics for public managers and policy analysts. Sage Publications.
Carli, R., Dotoli, M., Pellegrino, R., & Ranieri, L. (2013). Measuring and Managing the Smartness of Cities: A Framework for Classifying Performance Indicators. 2013 IEEE International Conference On Systems, Man, And Cybernetics. https://doi.org/10.1109/smc.2013.223
Conn, V. (2017). Don’t Rock the Analytical Boat: Correlation Is Not Causation. Western Journal Of Nursing Research, 39(6), 731-732. https://doi.org/10.1177/0193945917701090
Crato, N., & Paruolo, P. (2019). The Power of microdata. In N. Crato, & P. Paruolo (Eds.), Data-Driven Policy Impact Evaluation. How access to microdata is transforming policy design. (p. 1-14). Springer. http://doi.org/10.1007/978-3-319-78461-8
Deepa, N., Pham, Q., Nguyen, D., Bhattacharya, S., Prabadevi, B., & Gadekallu, T. et al. (2021). A Survey on Blockchain for Big Data: Approaches, Opportunities, and Future Directions. arXiv.org. https://arxiv.org/abs/2009.00858
Elia, L., Santangelo, G., & Schnepf, S. (2015). Synthesis report on the 'Pilot projects to carry out ESF related counterfactual impact evaluations. Directorate-General for Employment, Social Affairs and Inclusion (European Commission), Joint Research Centre (European Commission) https://op.europa.eu/en/publication-detail/-/publication/3d8a3354-084d-11e6-b713-01aa75ed71a1/language-en
European Commission. (2014, January). Monitoring and evaluation of European Cohesion Policy.European Regional Development Fund, European Social Fund, European Cohesion Fund. Guidance document on ex-ante evaluation. https://ec.europa.eu/regional_policy/sources/docoffic/2014/working/ex_ante_en.pdf
European Commission. (2019). Guidelines on linking planning/programming, monitoring and evaluation (2nd ed.). European Commission. https://ec.europa.eu/neighbourhood-enlargement/sites/default/files/dg_near_guidance_note_-_addressing_capacity_development_in_programming_me.pdf
European Commission. (2021). Evaluation. https://ec.europa.eu/regional_policy/en/policy/evaluations/member-states/
Ferraro, P. J. (2009). Counterfactual thinking and impact evaluation in environmental policy. New directions for evaluation, 2009(122), 75-84. https://doi.org/10.1002/ev.297
Field-Fote, E. (2019). Mediators and Moderators, Confounders and Covariates: Exploring the Variables That Illuminate or Obscure the “Active Ingredients” in Neurorehabilitation. Journal Of Neurologic Physical Therapy, 43(2), 83-84. https://doi.org/10.1097/npt.0000000000000275
Gertler, P., Martinez, S., Premand, P., Rawlings, L., & Vermeersch, C. (2016). Impact Evaluation in Practice, (2nd ed.). World Bank.
Hujer, R., Caliendo, M., & Radic, D. (2004). Methods and limitations of evaluation and impact research. Found Eval Impact Res (CEDEFOP Ref Ser 58), 30, 131-90. https://www.cedefop.europa.eu/files/BgR1_Hujer.pdf
Karafiloski, E., & Mishev, A. (2017). Blockchain solutions for big data challenges: A literature review. IEEE EUROCON 2017 – 17th International Conference On Smart Technologies. https://doi.org/10.1109/eurocon.2017.8011213
Li, Y., Li, P., Zhu, F., & Wang, R. (2017). Design of higher education quality monitoring and evaluation platform based on big data. 12Th International Conference On Computer Science And Education (ICCSE). https://doi.org/10.1109/iccse.2017.8085513
Lucaciu, L. (2018). A Look at the Evaluation Framework for Smart Growth Programmes. Revista Romaneasca Pentru Educatie Multidimensionala, 10(3), 60-76. https://doi.org/10.18662/rrem/63
Lucaciu, L., Roman, M., Rus, S., Florian, B., Vasile, M., & Mariș, C. (2020). Evaluarea retrospectivă a intervențiilor POSDRU 2007-2013 în domeniul educației. București: Ministerul Investițiilor și Proiectelor Europene.
Lucaciu, L., Roman, M., Rus, S., Florian, B., Vasile, M., & Mariș, C.(2021a). Prima evaluare a intervențiilor POCU 2014-2020 în domeniul incluziunii sociale [The first evaluation of the POCU interventions 2014-2020 in the field of social inclusion]. București: Ministerul Investițiilor și Proiectelor Europene.
Lucaciu, L., Roman, M., Savin, A., Ionescu, D., Bulai, A., Rus, S., Florian, B., Vasile, M., Maris, C., Buhaescu-Ciuca, S., Savin, A., Platon, G., Rosu, C., Perianu, E., Dalu, A.-M., Haj, M. C., Manoleli-Preda, D., Florescu, A.-E., Baciu, A., Jurj., A., Florina, T.(2021b). Prima evaluare a intervențiilor POCU 2014-2020 în domeniul educației [The first evaluation of the POCU interventions 2014-2020 in the field of education]. Bucharest: Ministerul Investițiilor și Proiectelor Europene. http://www.anc.edu.ro/wp-content/uploads/2021/04/Primul-raport-de-evaluare-a-interven%C8%9Biilor-POCU-2014-2020-%C3%AEn-domeniul-educa%C8%9Biei-1.pdf
Morris, S., Tödtling-Schönhofer, H., & Wiseman, M. (2013). Design and commissioning of counterfactual impact evaluations. https://op.europa.eu/en/publication-detail/-/publication/f879a9c1-4e50-4a7b-954c-9a88d1be369c
Popescu, M., & Roman, M. (2018). Vocational training and employability: Evaluation evidence from Romania. Evaluation And Program Planning, 67, 38-46. https://doi.org/10.1016/j.evalprogplan.2017.11.001
Rettore, E., & Trivellato, U. (2019). The Use of Administrative Data to Evaluate the Impact of Active Labor Market Policies: The Case of the Italian Liste di Mobilità. In N. Crato & P. Paruolo, Data-driven policy impact evaluation (1st ed., pp. 165-182). Springer Open
Roman, M., & Goțiu (Lucaciu), L. (2017). Non-parametric methods applied in the efficiency analysis of European structural funding in Romania. Journal Of Applied Quantitative Methods ,18(2). https://mpra.ub.uni-muenchen.de/80548/1/MPRA_paper_80548.pdf
Scott-Phillips, T., Dickins, T., & West, S. (2011). Evolutionary Theory and the Ultimate–Proximate Distinction in the Human Behavioral Sciences. Perspectives On Psychological Science, 6(1), 38-47. https://doi.org/10.1177/1745691610393528
Silverman, D. (2011). Interpreting qualitative data. A Guide to the Principles of Qualitative Research. Sage
Trivellato, U. (2019). Microdata for Social Sciences and Policy Evaluation as a Public Good. In N. Crato & P. Paruolo (Eds.), Data-Driven Policy Impact Evaluation (pp. 27-45). Springer Open.
United Nations. (2019). National Quality Assurance Frameworks Manual for Official Statistics. United Nations Department of Economic and Social Affairs Statistics Division. https://unstats.un.org/unsd/methodology/dataquality/references/1902216-UNNQAFManual-WEB.pdf.
Vaessen, J. (2011). Challenges in impact evaluation of development interventions: randomised experiments and complexity. Evaluating the complex: attribution, contribution, and beyond. comparative policy evaluation, 18, 283-313. https://medialibrary.uantwerpen.be/oldcontent/container2143/files/Publications/DP/2010/01-Vaessen.pdf
Wang, R., & Strong, D. (1996). Beyond Accuracy: What Data Quality Means to Data Consumers. Journal Of Management Information Systems, 12(4), 5-33. https://doi.org/10.1080/07421222.1996.11518099
White, H. (2014). Current challenges in impact evaluation. The European Journal of Development Research, 26(1), 18-30. https://doi.org/10.1057/ejdr.2013.45
Williams, B. (2015). Prosaic or profound? The adoption of systems ideas by impact evaluation. IDS Bulletin, 46(1), 7-16. https://doi.org/10.1111/1759-5436.12117
World Health Organization. (2017). Data quality review: module 1: framework and metrics. https://apps.who.int/iris/handle/10665/259224
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2021 The Authors & LUMEN Publishing House
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant this journal right of first publication, with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work, with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g. post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g. in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as an earlier and greater citation of published work (See The Effect of Open Access).
Postmodern Openings Journal has an Attribution-NonCommercial-NoDerivs
CC BY-NC-ND
