Today: Dec 27, 2024
RU / EN
Last update: Dec 27, 2024

Microbiological Oropharyngeal Patterns in Patients with Different Phenotypes of Chronic Obstructive Pulmonary Disease

Karnaushkina M.A., Fedosenko S.V., Sazonov A.E., Petrov V.A., Ovsyannikov D.Yu., Ogorodova L.M.

Key words: COPD; oropharyngeal microbiota; sequencing; COPD phenotypes; severity.

Persistent bronchial inflammation in chronic obstructive pulmonary disease (COPD) is considered the cause of ventilation disorders and related contamination with conditionally pathogenic microorganisms; the latter can proceed and transform into a full infection, which can aggravate and exacerbate COPD.

The aim of the study was to evaluate the relations between the oropharyngeal microbiota in patients with COPD and the clinical, functional, and prognostic parameters of the disease.

Materials and Methods. 64 patients with COPD were included in the study; the participants were scheduled to visit our clinic on two occasions. In the first visit, their medical history was studied in detail and the major examination procedures were conducted. Those included an assessment of the respiratory function, the 6-minute walk test, the degree of dyspnea by the Medical Research Council scale, body plethysmography, the diffusion capacity of the lungs, and a chest CT scan. The second visit took place 12 months after the first one to assess the changes in the course of the disease. The result was considered negative if, in the second examination, the patient‘s condition was found more severe. Oropharyngeal samples of all patients were sequenced to identify the V3–V4 variable sites of the 16S rRNA gene.

Results. It is found that the microbiological oropharyngeal patterns in COPD patients depend on the source of micro-aspiration. In addition, the changes in the oropharyngeal microbiota correlate with the severity and prognosis of the disease, as well as the patient phenotype. Based on the data obtained by sequencing parts of the 16S rRNA gene, the role of oropharyngeal microbiota in determining the course and prognosis of COPD has been elucidated.

Conclusion. The presented clinical and functional characteristics associated with oropharyngeal microbiota indicate that micro-aspirations from other body compartments not only affect the composition of oropharyngeal microbiota in patients with COPD but also have an important prognostic significance.


Introduction

The development of ventilation disorders in chronic obstructive pulmonary disease (COPD) is manifested by a progressive decrease in airflow associated with an abnormal inflammatory response of the lungs to harmful particles [1, 2], including microorganisms [1]. At the same time, chronic bronchopulmonary inflammation and a decreased airflow create conditions for contamination of the respiratory tract by opportunistic and pathogenic flora that change the composition of the respiratory microbiota [1, 3]. In recent publications, the bacterial contamination of the bronchial tree in COPD has been implicated in the increasing risk of COPD transformation into an infectious process with its exacerbations and disease progression [3, 4]. However, the involvement of the respiratory microbiota associated with the COPD phenotype in this process is still debatable.

Thus, Chambers et al. [5] demonstrated that the microbiota in the respiratory tract of patients with predominantly severe COPD significantly differed from the bacterial composition of the airways in healthy individuals. In the COPD patients, there were less Bacteroidesspp. and an increased representation of Firmicutes and Proteobacteria. Similar results were obtained by Charlson et al. [6]. It has been also suggested that the abnormal changes in microbiota could be initiated at the perinatal period. Thus, Górecka D. and Puścińska E. [7] analyzed the composition of the respiratory tract microbiota in a newborn and its correlation with baby’s nutrition, and the use of antibiotics by the mother. The authors suggested their research would shed light on the causes of chronic pulmonary disorders. However, none of the published studies analyzed the respiratory microbiota in respect to the disease progression and its clinical/functional parameters [8–11]. This may be due to the fact that, firstly, COPD is a clinically heterogeneous disease, and secondly, determination of the microbiota composition using classical microbiological methods does not provide a complete picture of its composition.

In this study, we used the phenotypic classification of COPD based on clinical and functional differences between the patients. As compared with the determination of the forced expiratory volume (FEV1) alone, the proposed parameter better reflects the severity and prognosis of the disease. Here, the composition of microbiota was determined using the methods of molecular-genetic identification of microorganisms [1, 4], which significantly changed our knowledge about the oropharyngeal microbiome. The obtained results enabled us to propose a prognostic model to predict the severity of COPD based on the taxonomic composition of the oropharyngeal microbiota.

The aim of the study was to evaluate the relations between the oropharyngeal microbiota in patients with COPD and the clinical, functional, and prognostic parameters of the disease.

Materials and Methods

This prospective non-interventional study included 64 patients with COPD. The average age was 57 (50–63) years.

All patients had confirmed COPD according to the GOLD 2015 criteria [1, 2]; the diagnosis was made at least 12 months before the enrollment, and their smoking index was higher than 10 packs/year. For 4 weeks prior to the enrollment, they received no antibiotic therapy.

The study was conducted in accordance with the Helsinki Declaration (2013) and approved by the local Ethics Committee of the Siberian State Medical University (protocol No.1927 of 28.03.2011). An informed consent has been obtained from each patient.

The study included two patient visits. During the first visit, all patients were clinically examined and their relevant medical history was carefully studied (below we will refer to these patients as the extended examination group). Tolerance to exercise was determined using a 6-minute walk test. The degree of dyspnea was assessed by the MRC (Medical Research Council) scores; the external respiration was evaluated using the bronchodilation test and the pulmonary diffusion index (body plethysmography). The study was performed using a MasterScreen Body instrumentation (Erich Jaeger, Germany). The obtained data were compared to the reference values calculated by the generally accepted formulas recommended by the European Respiratory Society and American Thoracic Society. All patients underwent computed tomography scan of the chest (chest CT) with a Somatom Sensation 40 spiral CT scanner (Siemens, Germany). Densitometry of the lung tissue and bronchial structure was performed using the Pulmo CT program (Siemens, Germany). During the first visit, oropharyngeal swabs were taken from all patients.

Based on the clinical and functional parameters obtained at the examination, the patients were divided into groups according to three phenotypes of COPD: “chronic bronchitis” (n=30), “emphysema” (n=14), and “mixed” (n=20). This stratification was based on the emphysema and bronchitis phenotypes as proposed by Burrows (1966) and Dornhorst (1955) and modified by us.

The phenotype of “chronic bronchitis” describes patients whose major clinical symptom (aside from exacerbation periods) was cough with mucopurulent sputum. At auscultation, harsh breathing sounds and rales were detected; the chest CT showed signs of bronchitis, the diffusion lung capacity was normal.

The “emphysema” phenotype refers to patients whose major clinical symptom (aside from exacerbation periods) was shortness of breath; cough was mild and unproductive. At auscultation, a weakened vesicular breath with wheezing episodes was heard; the characteristic signs and a decreased diffusion capacity were present in the CT scans.

Patients, who showed no obvious signs of bronchitis or emphysema during non-exacerbation periods, were included in the “mixed” phenotype group.

During the second visit, which took place 12 months after the first one, all patients underwent a clinical and functional examination, an assessment of the exacerbation rate, an evaluation of dyspnea by the MRC scale, a determination of exercise tolerance and a test of external respiration using a bronchodilator probe. We analyzed the dynamics of COPD during the past 12 months of observation, taking into account the severity of the symptoms, the post-bronchodilation value of FEV1 and the risk of exacerbations (GOLD 2015). The dynamics of the disease was considered negative if, at the second examination, the patient presented with a more severe course of COPD than he did on the first visit. The oropharyngeal samples were not taken during the second visit.

The Statistica 10.0 software package was used for statistical processing of the medical history data and the clinical/functional indices. When comparing the qualitative characteristics, the criterion χ2 was applied. To calculate the difference of the means between non-paired samples, the Mann–Whitney U test was used. The qualitative data are presented as absolute or relative (%) values. The quantitative data are presented as follows: those with a non-normal distribution — as Me [Q25; Q75]; and those with the normal distribution — as M±SD, where M is the arithmetic mean, SD is the standard deviation. The normality of distributions of was tested with the χ2 criterion. The difference of the means was considered statistically significant at p<0.05. The Spearman coefficient was used for the correlation analysis. The strength of the correlation was denoted as strong at ±0.7 to ±1; average at ±0.3 to ±0.699; and weak at 0 to ±0.299.

Samples of oropharyngeal swabs of all patients were tested for the presence of the 16S rRNA gene and the V3–V4 sites sequences according to the 16S Metagenomic Sequencing Library Preparation protocol recommended by Illumina (USA) for the MiSeq sequencer. A taxonomic classification was carried out according to the sequence database of the 16S rRNA gene of Greengenes v. 13.5 with the help of the Bayesian classifier and the QIIME (Quantitative Insights into Microbial Ecology) software. To find taxonomic differences between the phenotype-specified groups of samples at the level of the operating taxonomic unit (OTU), a method based on linear regression analysis and adjusted to the metagenome data by the metagenomeSeq package was used [12]. To detect clusters of microorganisms, the CoNet plugin version 1.1.beta [13] of the Cytoscape software was utilized. Correlations between the amounts of individual microorganisms were determined according to the Spearman method, only the correlations with a coefficient of 0.05 or higher were considered. To quantify the impact of the oropharyngeal microbiota on COPD progression, a prognostic model was developed using the caret package, where the recursive elimination of variables with a 10-fold cross-validation was applied.

Results and Discussion

In this study, a comparative intergroup analysis of the clinical and functional parameters of patients with different COPD phenotypes was performed. As a result, clinically important data were obtained to suggest that patients with the “mixed” phenotype of COPD had the most severe manifestations of the disease. Thus, in comparison with the two other phenotypes, “chronic bronchitis” and “emphysema”, patients with the “mixed” phenotype had more exacerbations per year and a higher value of the BODE index. The FEV1 values for the “mixed” phenotype were significantly lower than those for the two other phenotypes (see the Table).


karnaushkina-table.jpg

Comparative characteristics of patients (the extended examination group) with different phenotypes of chronic obstructive pulmonary disease (Me [Q1; Q3])


The analyzed data indicated that patients with the “mixed” COPD phenotype were treated with inhalations recommended for patients with severe and extremely severe COPD: long-acting anticholinergic drugs + long-acting β2-agonists + inhaled glucocorticoids. In this group, the number of antibacterial therapy courses during the year was significantly higher as compared with the two other groups.

The composition of oropharyngeal microbiota in patients with different clinical and functional types of COPD was dominated by the Streptococcus, Veillonella, and Prevotella genera of microorganisms (10% and more) in all phenotypes of the disease (Figure 1). Our results on these “highly represented taxa” agree with the data reported by others [14–17].


karnaushkina-fig-1.jpg Figure 1. Prevalent genera of microorganisms, comprising 95% of the metagenome of the oropharyngeal microbiota in patients with chronic obstructive pulmonary disease (the extended examination group)

The metagenomeSeq-assisted analysis of low-represented taxa (less than 10% of the identified microbiome) revealed a predominance of bacterial commensals in patients with the phenotype of emphysema. Among those, the Firmicutes bacteria (Acidaminococcus LogFC 1.865; Anaerovibrio LogFC 2.653; Leuconostoc LogFC 1.727; Oceanobacillus LogFC 1.107; Phascolarctobacterium LogFC 2.915; Tetragenococcus LogFC 1.513), the Actinobacteria bacteria (Micrococcus LogFC 1.002; Dermacoccus LogFC 2.191), and the Proteobacteria bacteria (Cardiobacterium LogFC 1.002; Novosphingobium LogFC 1.561) (p<0.001).

Analysis of the low-represented taxa in patients with the “chronic bronchitis” phenotype revealed a significant predominance of bacteria of Fusobacterium, Actinobacteria, Proteobacteria, Bacteroidetes, Firmicutes, Bacteroides, mainly due to a higher content of opportunistic bacteria of Haemophilus (LogFC 8.662), Actinomyces (LogFC 9.813), Lactobacillus (LogFC 4.758), Leptotrichia (LogFC 10.556), Porphyromonas (LogFC 5.896), Bacillus (LogFC 3.161), Gemella (LogFC 8.303) (p<0.001). Also in these patients, a considerable amount of opportunistic bacteria in the oropharynx was found: Prevotella melaninogenica species (LogFC 11.738), Rothia mucilaginosa (LogFC 9.257), and Veillonella parvula (LogFC 10.631) (p<0.001).

Samples of oropharyngeal swabs from patients with COPD with the “mixed” phenotype showed the prevailing representation of the Firmicutes and Bacteroides bacteria (p<0.001) with a less diverse variety of species in comparison with the two other phenotypes.

To identify the respiratory microbiota patterns associated with specific clinical and functional characteristics of the COPD patients, the Spearman’s correlation analysis was performed.

In the “chronic bronchitis” phenotype patients, a cluster of conditionally pathogenic bacteria was found; within the cluster, a strong positive correlation between the different microbial taxa was identified: Fusobacterium + Prevotella + Mycoplasma + Bulleidia (r=0.701; p<0.05).

In the COPD patients with the “emphysema” phenotype, a cluster of normal oropharyngeal microbiota of the Firmicutes type was found. In addition, an Actinobacteria cluster of opportunistic bacteria of the Enterococcus gastrointestinal type and strong positive intra-cluster correlations was also present (r>0.08, p<0.001 and r>0.08, p<0.05, respectively).

In the “mixed” phenotype, two clusters of opportunistic oropharyngeal bacteria with a strong positive correlation were found: Porphyromonas + Fusobacterium + Prevotella (r=0.832, p<0.05) and Streptococcus + Rothia (r=0.761, p<0.05).

Thus, the patients with COPD showed three patterns of oropharyngeal microbiota composition. In patients with the “chronic bronchitis” phenotype, opportunistic bacteria were mostly represented. In patients with the “emphysema” phenotype, the oropharyngeal normobiota and intestinal microbiota (including conditionally-pathogenic bacteria) were dominant. In patients with the “mixed” phenotype, a pattern with a low species diversity and a predominance of gastrointestinal opportunistic microbiota (Firmicutes), in combination with the opportunistic bacteria cluster (Porphyromonas + Prevotella + Fusobacterium) was found.

In this study, it was attempted, for the first time, to develop a prognostic model based on the clinical and functional phenotypes of COPD and their respective microbiological patterns. Our analysis indicates that an increase in the oropharyngeal Moraxellaceae, Peptococcaceae, Eubacteriaceae, Enterococcaceae, Staphylococcaceae, and Actinomycetaceae bacterial families is an unfavorable prognostic factor in terms of the disease progression. In the validation test, this model showed a sensitivity of 0.67 and specificity of 1, a prognosis accuracy of 75% with an area under the ROS curve of 0.72 (Figure 2).


karnaushkina-fig-2.jpg Figure 2. Multiple correlations between different components of the oropharyngeal microbiota in patients with different prognoses of chronic obstructive pulmonary disease
The data includes variables with a Spearman correlation value of
≥0.05 at p<0.05 after introducing the Benjamin–Hochberg correction for multiple comparisons. The blue diamond on the chart shows the optimal sub-selection of variables, which allows to classification of the patients with an accuracy of 76%


Thus, it can be argued that certain changes in the composition of oropharyngeal microbiota are significant for the progression of COPD.

To interpret the obtained results, we assumed that the composition of the oropharyngeal microbiota depends on the source of micro-aspiration (gastrointestinal tract, nasopharynx, oral cavity). Numerous studies [15, 18, 19] demonstrated the role of opportunistic oropharyngeal bacteria in the development of pneumonia and exacerbations of chronic bronchopulmonary diseases. These bacteria enter the lower respiratory tract as a result of micro-aspiration from the upper respiratory tract. According to our study, 40% of patients with the “chronic bronchitis” phenotype and 70% with the “mixed” phenotype suffered from tonsillitis, sinusitis or periodontitis. In these patients, we detected the formation of clusters of oropharyngeal opportunistic bacteria: Fusobacterium + Prevotella + Mycoplasma + Bulleidia, Porphyromonas + Prevotella + Fusobacterium and Porphyromonas + Prevotella + Streptococcus minor. This finding supports the hypothesis that the aspirated oropharyngeal microflora transforms the composition of the airways microbiome in patients with the “mixed” and “chronic bronchitis” phenotypes of COPD.

High levels of bacteria typical for the gastrointestinal microbiota were found in oropharyngeal swabs from patients with the “emphysema” phenotype; that included a cluster of normal oropharyngeal microflora (such as Firmicutes) and Actinobacteria + Enterococcus (opportunistic gastrointestinal microbiota). Published [20, 21] studies have demonstrated that gastroesophageal reflux disease (GERD) is associated with chronic bronchopulmonary diseases, including COPD, thus aggravating the course of the main disease. In our study, GERD was found in 11 patients (78.6%) with the phenotype of “emphysema”, in 15 patients (75%) with the “mixed” phenotype, and only 6 patients (20%) with the phenotype of “chronic bronchitis”. It is known [14–17] that the presence of the Porphyromonas and Prevotellgenera pathogenic bacteria detected in patients with the “mixed” phenotype is clinically significant because this is associated with purulent inflammatory processes. Likewise, the impact of the respiratory tract opportunistic microflora on the pulmonary tissue is enhanced by the gastrointestinal microbiota that enters the airways. The low diversity of bacterial species in this COPD phenotype decreases their resistance to pathogens [17] and can severely complicate the course of COPD in patients with the “mixed” phenotype.

The presented clinical and functional characteristics of COPD patients and their correlations with the composition of the oropharyngeal microbiota allow us to propose that micro-aspiration from different body compartments affects not only the oropharyngeal microbiota but also contributes to the phenotypic characterization and clinical prognosis of COPD. At present, it is difficult to ascertain, what are the primary factors determining the clinical and functional phenotype of COPD. They can be either the phenotypic traits of the disease (e.g., the impact of GERD) or a disorder of the upper respiratory tract or the composition of the oropharyngeal microbiota. Reasonably, a combination of the above factors should be considered.

As shown in the present study, the low species diversity of the oropharyngeal microbiota in patients with COPD of the “mixed” phenotype may favor progression and aggravation of the disease.

Conclusion

Each phenotype of chronic obstructive pulmonary disease with its specific clinical and functional features is characterized by a specific pattern of oropharyngeal microbiota, which determines the severity and prognosis of the disease. The bacterial composition of these patterns depends on the source of micro-aspiration (upper gastrointestinal tract or nasopharynx). In addition, the composition of the oropharyngeal microbiota is determined by the severity of the disease and by its phenotype. This phenotype determines the course of the disease in a patient with chronic obstructive pulmonary disease.

Financial support. The study was supported by a grant from the Federal target program “Research and development in priority areas for the development of the scientific and technological complex in Russia for 2014–2020” (GC No.14.604.21.0075, unique identifier RFMEFI60414X0075).

Conflict of interest. The authors confirm the absence of financial and other conflicting interests that could influence their work.


References

  1. Chuchalin A.G., Avdeev S.N., Aysanov Z.R., Belevskiy A.S., Leshchenko I.V., Meshcheryakova N.N., Ovcharenko S.I., Shmelev E.I. Russian respiratory society. Federal guidelines on diagnosis and treatment of chronic obstructive pulmonary disease. Russian Pulmonology 2014; 3: 15–36, https://doi.org/10.18093/0869-0189-2014-0-3-15-54.
  2. Global Initiative for Chronic Obstructive Lung Disease. Global strategy for diagnosis, management, and prevention of COPD. Updated 2015.
  3. Fedosenko S.V., Ogorodova L.M., Popenko A.S., Petrov V.A., Tyakht A.V., Saltykova I.V., Kamaltynova E.M., Deev I.A., Kulikov E.S., Kirillova N.A., Govorun V.M., Kostryukova E.S., Karnaushkina M.A. The qualitative and quantitative characteristics of content of oro-pharyngeal microbiota of patients with chronic obstructive disease of lungs. Lechenie i profilaktika 2015; 2(14): 92–99.
  4. Avdeev S.N. Symptoms and quality of life in patients with chronic obstructive pulmonary disease: are there patient-defined outcomes or predictive factors? Russian Pulmonology 2016; 26(2): 231–237, https://doi.org/10.18093/0869-0189-2016-26-2-231-237.
  5. Chambers D.C., Gellatly S.L., Hugenholtz P., Hansbro P.M. JTD special edition ‘Hot Topics in COPD’ — the microbiome in COPD. J Thorac Dis 2014; 6(11): 1525–1531, https://doi.org/10.3978/j.issn.2072-1439.2014.11.08.
  6. Charlson E.S., Bittinger K., Haas A.R., Fitzgerald A.S., Frank I., Yadav A., Bushman F.D., Collman R.G. Topographical continuity of bacterial populations in the healthy human respiratory tract. Am J Respir Crit Care Med 2011; 184: 957–963, https://doi.org/10.1164/rccm.201104-0655oc.
  7. Górecka D., Puścińska E. Microbiome of the lung. Pneumonol Alergol Pol 2014; 82(6): 481–485, https://doi.org/10.5603/PiAP.2014.0063.
  8. Park H., Shin J.W., Park S.G., Kim W. Microbial communities in the upper respiratory tract of patients with asthma and chronic obstructive pulmonary disease. PLoS One 2014; 9(10): e109710, https://doi.org/10.1371/journal.pone.0109710.
  9. Molyneaux P.L., Mallia P., Cox M.J., Footitt J., Willis-Owen S.A., Homola D., Trujillo-Torralbo M.B., Elkin S., Kon O.M., Cookson W.O., Moffatt M.F., Johnston S.L. Outgrowth of the bacterial airway microbiome following rhinovirus exacerbation of chronic obstructive pulmonary disease. Am J Respir Crit Care Med 2013; 188(10): 1224–1231, https://doi.org/10.1164/rccm.201302-0341oc.
  10. Patel A.R., Hurst J.R. Extrapulmonary comorbidities in chronic obstructive pulmonary disease: state of the art. Expert Rev Respir Med 2011; 5(5): 647–662, https://doi.org/10.1586/ers.11.62.
  11. Hurst J.R. Upper airway. 3: Sinonasal involvement in chronic obstructive pulmonary disease. Thorax 2010; 65(1): 85–90, https://doi.org/10.1136/thx.2008.112888.
  12. Paulson J.N., Olson N.D., Wagner J., Talukder H., Pop M., Bravo H.C. Statistical analysis for sparse high-throughput sequencing, https://doi.org/10.18129/B9.bioc.metagenomeSeq.
  13. Faust K., Sathirapongsasuti J.F., Izard J., Segata N., Gevers D., Raes J., Huttenhower C. Microbial co-occurrence relationships in the human microbiome. PLoS Comput Biol 2012; 8(7): e1002606, https://doi.org/10.1371/journal.pcbi.1002606.
  14. Pragman A.A., Kim H.B., Reilly C.S., Wendt C., Isaacson R.E. The lung microbiome in moderate and severe chronic obstructive pulmonary disease. PloS One 2012; 7(10): e47305, https://doi.org/10.1371/journal.pone.0047305.
  15. Sze M.A., Dimitriu P.A., Hayashi S., Elliott W.M., McDonough J.E., Gosselink J.V., Cooper J., Sin D.D., Mohn W.W., Hogg J.C. The lung tissue microbiome in chronic obstructive pulmonary disease. Am J Respir Crit Care Med 2012; 185(10): 1073–1080, https://doi.org/10.1164/rccm.201111-2075oc.
  16. Sin D., Sze M., Hogg J. Bacterial microbiome of lungs in COPD. Int J Chron Obstruct Pulmon Dis 2014; 9: 229–238, https://doi.org/10.2147/copd.s38932.
  17. Huang Y.J., Sethi S., Murphy T., Nariya S., Boushey H.A., Lynch S.V. Airway microbiome dynamics in exacerbations of chronic obstructive pulmonary disease. J Clin Microbiol 2014; 52(8): 2813–2823, https://doi.org/10.1128/jcm.00035-14.
  18. Ghannoum M.A., Jurevic R.J., Mukherjee P.K., Cui F., Sikaroodi M., Naqvi A., Gillevet P.M. Characterization of the oral fungal microbiome (mycobiome) in healthy individuals. PLoS Pathog 2010; 6(1): e1000713, https://doi.org/10.1371/journal.ppat.1000713.
  19. Hsiao W.W., Li K.L., Liu Z., Jones C., Fraser-Liggett C.M., Fouad A.F. Microbial transformation from normal oral microbiota to acute endodontic infections. BMC Genomics 2012; 13: 345, https://doi.org/10.1186/1471-2164-13-345.
  20. Pei Z., Yang L., Peek R.M., Jr Levine S.M., Pride D.T., Blaser M.J. Bacterial biota in reflux esophagitis and Barrett’s esophagus. World J Gastroenterol 2005; 11(46): 7277–7283, https://doi.org/10.3748/wjg.v11.i46.7277.
  21. Sakae T.M., Pizzichini M.M., Teixeira P.J., Silva R.M., Trevisol D.J., Pizzichini E. Exacerbations of COPD and symptoms of gastroesophageal reflux: a systematic review and meta-analysis. J Bras Pneumol 2013; 39(3): 259–271, https://doi.org/10.1590/s1806-37132013000300002.


Journal in Databases

pubmed_logo.jpg

web_of_science.jpg

scopus.jpg

crossref.jpg

ebsco.jpg

embase.jpg

ulrich.jpg

cyberleninka.jpg

e-library.jpg

lan.jpg

ajd.jpg

SCImago Journal & Country Rank