Mapping output in content analysis in SciELO Brazil indicates a technique that is caught in a time warp

Rafael Cardoso Sampaio, Associate Professor of the Department of Political Science at the Universidade Federal do Paraná, Curitiba, Paraná, Brazil.

Diógenes Lycarião, Associate Professor of Journalism of the Institute of Culture and Art at the Universidade Federal do Ceará, Fortaleza, Ceará, Brazil.

The article Mapping and reflections on the use of content analysis in SciELO-Brazil (2002-2019)^1,2 has just been published in the Portuguese journal New Trends on Qualitative Research, indexed in SciELO Portugal. This is a scientometric review on the use of content analysis, in all research areas of SciELO Brazil, between the years 2002 and 2019 (SAMPAIO et al, 2022a).

Content analysis (CA) can be considered one of the main forms of qualitative data analysis in Brazilian research, either for analyzing texts and contents produced by different institutions or actors, or for evaluating the results generated by other data collection techniques of qualitative methods, such as interviews and focus groups, being also integrated with other techniques, such as systematic literature review (LYCARIÃO et al, 2022).

Image: Freepik.

We searched the SciELO collection in the Web of Science database and, after manually cleaning articles that did not use the technique, we reached a total of 3,484 references. In order to better understand the results, we divided the articles according to the three disciplines of the Coordination for the Improvement of Higher Education Personnel (Coordenação de Aperfeiçoamento de Pessoal de Nível Superior, CAPES): Life Sciences, Humanities, and Exact, Technological and Multidisciplinary Sciences.

Using the bibliographic coupling technique, through the VOSviewer software, our data indicate that there is more output in the Life Sciences discipline (n= 2,087) than in the Humanities discipline (n= 1,211), including highlights for journals in these areas, notably Collective Health and Nursing. While it sounds natural, it is the first time that this has been comprehensively evidenced in Brazilian research. As we know, the modern version of the scientific technique was created by Harold Laswell to analyze war propaganda in the 1920s, and then adopted by political communication later on.

Image: Sampaio et al (2022a).

Figure 1. CA-based publications in Brazilian journals

Now, our main finding confirms what we already suspected: Laurence Bardin’s manual “Content Analysis” concentrates the vast majority of citations. To reach this conclusion, we made a network of co-citations in our corpus, using VOSViewer (ECK, WALTMAN, 2010). All the clusters formed by the software show Bardin’s undisputed predominance in the Life Sciences and Humanities disciplines. We discuss below the relevance of such a long-awaited result.

Image: Sampaio et al (2022a).

Figure 2. Aggregate citations in the Life Sciences discipline

Image: Sampaio et al (2022a).

Figure 3. Humanities co-citation network graph.

As said, any researcher working with content analysis in Brazil knows (and usually points to) Bardin’s manual, so the fact that it is often cited in all areas is not surprising in itself. What is our concern then? Although several studies have evaluated content analysis in their respective fields of activity (NASCIMENTO, et al., 2021; CASTRO, et al., 2011; PALMEIRA, et al., 2020), as stated, there was no comprehensive survey of all areas of research on the matter. Now, empirically, it has been demonstrated that there are many citations to Bardin by all fields of knowledge in Brazil.

Then the real questions begin. What’s wrong with Bardin being much cited? In summary, the manual was published in 1977 and last updated in the early 1990s, which means that it does not reflect the main methodological and epistemological discussions of the last 30 years. Some possible consequences are discussed in the text:

• If used for quantitative CA, the manual does not cover in detail more sophisticated procedures such as creating a codebook, advanced text sampling techniques, coder training, reliability testing, and after coding statistical tests.
• If used for a qualitative CA, the manual does not satisfactorily address the creation of a qualitative corpus, the elaboration of the codebook (if it is a deductive analysis), the way in which the subjectivity of coders/judges was dealt with, methodological transparency for increased research credibility and forms of validation of qualitative data.
• Still in the qualitative area, CAQDAS software such as Atlas.Ti, MAXQDA, Nvivo, and WEBQDA, which were in their first editions in the early 90s, are more strongly recommended for use in qualitative research.
• There has been a significant evolution in the ethical debate about how to treat the results of analysis, especially when used to analyze content generated by subject collection techniques such as in-depth interviews, focus groups, and participant observation.
• In the last update of the Bardin manual, even the commercial Internet had not spread significantly. And the impacts of digital media on academic research in terms of access, visibility, impact, and collaboration are extremely relevant.
• Just as an example, the advent of online repositories presents a revolution in the conditions of research transparency and replication, since sharing original materials (such as medical records, journals, videos, transcripts of interviews, and focus groups etc.), as well as supplementary materials (the codebook or even the coded database itself, for example) entailed high costs.

Finally, our study indicates that the situation is not improving. Or the results don’t just reflect the aggregated data. When checking citations longitudinally, the use of different Bardin manuals grows over the years. Below, we present the graph showing the use of content analysis in SciELO Brazil and the number of citations to Bardin, both on a year basis. It is notable how both show remarkably similar growth curves.

Image: Sampaio et al (2022a).

Figure 4. Comparison between the CA output and citation to Bardin.

In other words, our data does not indicate the appearance or growth of other manuals or even texts on content analysis in Brazil and, on the contrary, indicates a growth in the use of Bardin’s manual that follows the use of content analysis in Brazil. In practice, when looking at the graphs, the impression we have is that content analysis is synonymous with Bardin in national research.

In short, while there seems to be an expanded epistemological and methodological discussion about the scientificity of qualitative methods, especially in terms of sample, analysis saturation, transparency, data validity etc. (e.g. BICUDO; COSTA, 2019; LIMA; RAMOS, PAULA, 2019), it seems not to occur when it comes to content analysis, which seems to remain stuck in a manual that has not been updated for over two decades.

As already warned by some authors (e.g., GONDIM; BENDASSOLLI, 2014; NASCIMENTO et al, 2021; CASTRO et al, 2011), content analysis seems to be used in an ad hoc, uncritical way and often just to justify the existence of the name of a search technique. We fear that this may be encouraging low rigor and low-quality research or, as we concluded in a second evaluation, there is a lot of Bardin, but little quality (SAMPAIO, et al., 2022b). In addition, we believe that the way in which the technique is used may be frozen in time.

The purpose of the text is precisely to initiate the discussion and denaturalize this inadvertent use of Bardin’s manual, at the same time that we further develop these discussions and present other manuals that are references in the specialized international literature.

Notes

1. The original version of this study was published in SciELO Preprints and is available from: SAMPAIO, R.C., et al. A technique stuck in time? Mapping scientific production based on content analysis at SciELO Brazil (2002-19). SciELO Preprints [online]. 2021 [viewed 5 May 2023]. https://doi.org/10.1590/SciELOPreprints.1913. Available from: https://preprints.scielo.org/index.php/scielo/preprint/view/1913

2. SAMPAIO, R., et al. Mapeamento e reflexões sobre o uso da análise de conteúdo na SciELO-Brasil (2002-2019). New Trends in Qualitative Research [online]. 2022a, vol. 15, e747 [viewed 3 May 2023]. https://doi.org/10.36367/ntqr.15.2022.e747. Available from: http://www.scielo.pt/scielo.php?script=sci_arttext&pid=S2184-77702022000600011&lang=pt

References

BICUDO, M.A.V., and COSTA, A.P. (ed.) Leituras em pesquisa qualitativa. São Paulo: LF Editorial, 2019.

CASTRO, T.G., ABS, D. and SARRIERA, J.C. Análise de conteúdo em pesquisas de Psicologia. Psicol. cienc. prof. [online]. 2021, vol. 31, no. 4, pp. 814-825 [viewed 3 May 2023]. https://doi.org/10.1590/s1414-98932011000400011. Available from: https://www.scielo.br/j/pcp/a/FT97F8CvRpQLF3W46vTdK8d/

ECK, N.J. and WALTMAN, L. Software survey: VOSviewer, a computer program for bibliometric mapping. Scientometrics [online]. 2010, vol. 84, no. 2, pp. 523-538 [viewed 3 May 2023]. https://doi.org/10.1007/s11192-009-0146-3. Available from: https://link.springer.com/article/10.1007/s11192-009-0146-3

GONDIM, S.M.G. and BENDASSOLLI, P.F. Uma crítica da utilização da análise de conteúdo qualitativa em psicologia. Psicol. Estud. [online]. 2014, vol. 19, no. 2, pp. 191-199 [viewed 3 May 2023]. https://doi.org/10.1590/1413-73722053000. Available from: https://www.scielo.br/j/pe/a/M6cwvMCxw8xnKcbMVhmTBmp/

LIMA, V.M.R., RAMOS, M.G. and PAULA, M.C. (ed.) Métodos de análise em pesquisa qualitativa: releituras atuais. Porto Alegre: EdiPUCRS, 2019.

LYCARIÃO, D., ROQUE, R. and COSTA, D. Revisão Sistemática de Literatura (RSL) e Análise de Conteúdo (AC) na Área na Comunicação e Informação: o problema da confiabilidade e como resolvê-lo. (Submetido à publicação), 2022.

NASCIMENTO, O.A.S., et al. The use of Content Analysis in scientific production of Brazilian Physical Education. Rev. Bras. Ciênc. Esporte [online]. 2021, vol. 43, e006521 [viewed 3 May 2023]. https://doi.org/10.1590/rbce.43.e006521. Available from: https://www.scielo.br/j/rbce/a/n6j9LnWHGfGjTZtKVnsy8mp/

PALMEIRA, L.L., CORDEIRO, C.P.B.S. and PRADO, E.C. A análise de conteúdo e sua importância como instrumento de interpretação dos dados qualitativos nas pesquisas educacionais. Cadernos de Pós-graduação [online]. 2020, vol. 19, no. 1, pp. 14–31. ISSN: 2525-3514 [viewed 3 May 2023]. https://doi.org/10.5585/cpg.v19n1.17159. Available from: https://periodicos.uninove.br/cadernosdepos/article/view/17159

SAMPAIO, R.C. No Brasil, Análise de Conteúdo é sinônimo de Bardin e porque isso é um problema. Instituto Brasileiro de Pesquisa e Análise de Dados [online]. 2021 [viewed 3 May 2023]. Available from: https://ibpad.com.br/ciencia-dados/no-brasil-analise-de-conteudo-e-sinonimo-de-bardin-por-que-isso-e-um-problema/

SAMPAIO, R.C., et al. A technique stuck in time? Mapping scientific production based on content analysis at SciELO Brazil (2002-19). SciELO Preprints [online]. 2021 [viewed 5 May 2023]. https://doi.org/10.1590/SciELOPreprints.1913. Available from: https://preprints.scielo.org/index.php/scielo/preprint/view/1913

SAMPAIO, R.C., et al. Muita Bardin, pouca qualidade: uma avaliação sobre as análises de conteúdo qualitativas no Brasil. Revista Pesquisa Qualitativa [online], 2022b, vol. 10, no. 25, pp. 464-494 [viewed 3 May 2023]. https://doi.org/10.33361/RPQ.2022.v.10.n.25.547. Available from: https://editora.sepq.org.br/rpq/article/view/547

To read the article, access

SAMPAIO, R.C., et al. Mapping and reflections on the use of content analysis in SciELO-Brazil (2002-2019). New Trends in Qualitative Research [online]. 2022a, vol. 15, e747 [viewed 3 May 2023]. https://doi.org/10.36367/ntqr.15.2022.e747. Available from: http://www.scielo.pt/scielo.php?script=sci_arttext&pid=S2184-77702022000600011&lang=pt

External links

New Trends in Qualitative Research – NTQR: http://scielo.pt/scielo.php?script=sci_serial&pid=2184-7770&lng=pt&nrm=iso

VOSviewer: https://www.vosviewer.com/

About Rafael Cardoso Sampaio

Professor at the Department of Political Science at the Universidade Federal do Paraná (UFPR) and permanent professor at the Graduate Programs in Political Science and Communication at UFPR. He is leader of the research group on Political Communication and Digital Democracy (COMPADD). A CNPQ-2 productivity scholarship holder and researcher at the National Institute of Science and Technology in Digital Democracy (Instituto Nacional de Ciência e Tecnologia em Democracia Digital, INCT.DD). He is the author of the book “Análise de conteúdo categorial: manual de aplicação” published by the Escola Nacional de Administração Pública (ENAP).

About Diógenes Lycarião

Associate Professor of Journalism at the Institute of Culture and Art at the Universidade Federal do Ceará (UFC), and permanent researcher at the Graduate Program in Communication. He is leader of the Policy, Public Opinion and Communication Research Group (Gruppocom). He is the executive secretary of Compolítica (Brazilian Association of Researchers in Communication and Politics), member of the National Institute of Science and Technology in Informational Disputes and Sovereignties (Instituto Nacional de Ciência e Tecnologia em Disputas e Soberanias Informacionais, INCT/DSI) and vice-coordinator of the WG Communication of Science and Scientific Policies of Compós (Brazilian Association of Graduate Programs in Communication). He is the author of the book “Análise de conteúdo categorial: manual de aplicação” published by ENAP.

Translated from the original in Portuguese by Lilian Nassi-Calò.