Apresentação realizada em Oslo - Noruega, 2015.

1. A tool to monitor and evaluate a HTA network: the case of REBRATS
Leão,L.S.C.1; Vanni,T.1
1 Department of Science and Technology, Brazilian Ministry of Health
INTRODUCTION
After six years of existence, The Brazilian Network for Health
Technology Assessment (REBRATS) now encompasses 78 participating
institutions. Since its creation, there is a growing need to certificate and to
monitor the activities of its members.
The creation of a methodology is required in order to monitor the
network’s production and collaboration process. An annual certification of
the institutions through an activity report aims to fill this gap.
This process would also contribute to the promotion of effective
participation of member institutions and to greater collaboration between
them.
The study of scientific collaboration
networks seeks to correlate attributes of its
forming entities, patterns of relationships
between then and network performance. In a
study done on SISREBRATSi a "small world
phenomenon" was observed, suggesting that
most of these collaborations occurs in a small
circle of researchers. Therefore there is space to
better integrate the network in order to generate
higher economy of scale, expertise, knowledge
transfer, and to reduce the duplication of work.
However, SISREBRATSi data analysis is
limited since it refers only to the included
studies, which are a fraction of the total
production of the network institutions. Besides
that it does not capture other types of
collaborations like trainings and events. Thus it is
necessary to conduct primary data collection
related to these collaborations.
iREBRATS’ studies data base.
BACKGROUND
Figure 2. REBRATS’ sociogram based on
coauthorship evaluation of SISREBRATS’
studies.
Figure 1. REBRATS’ website: rebrats.saude.gov.br

2. A tool to monitor and evaluate a HTA network: the case of REBRATS.
Leão,L.S.C.1; Vanni,T.1
1 Department of Science and Technology, Brazilian Ministry of Health
In order to define a model of primary data collection relating to HTA
collaborations, a systematic review was conducted on MEDLINE . The consultation
yielded 260 records. By screening titles and abstracts, we identified 8 eligible
studies. During the analysis of the methodologies used in these eight articles,
important aspects related to the purpose of the study, the instrument used, the
collection strategy and the response rate were identified.
Search strategy: ("network analysis"[All Fields] OR "network structure"[All Fields]
OR "network theory"[All Fields]) AND ("data collection"[All Fields] OR "primary
data"[All Fields] OR "data-collection"[All Fields] OR "data gathering"[All Fields] OR
"data-gathering"[All Fields] OR "survey"[All Fields])
RESULTS
Katerndahl (2012) demonstrated a high response rate (100%) when
performing data collection along with the institutional annual report that all
members of the institution needed to fill. This strategy also guaranteed the
longitudinal data collection for 13 years. Kossinets (2006) states that a response
rate between 50% to 70% is acceptable, since it is unlikely to affect the test results
(Grosser, Lopez-Kidwell, & LaBianca, 2010). Although this is a common problem for
many network analysis studies with primary data collection, there is no consensual
solution. The solutions ranging from substitution with symmetrical knots
(Huisman, 2009; Steglich & Huisman, 2008; Stork & Richards, 1992) to more robust
methods for Bayesian inference (Butts, 2003).
There are different types of collaboration that must be evaluated
separately. Like the study of Okamoto et al (2015) which considered six types of
collaboration: study or research protocol, co-authored publication, co-authored
presentation, mentoring or training, committee / working group and others. The
category "other" was included for quality control purposes in order to get a general
measure of global collaboration. This study also considered the current and
previous collaborations network formation. Mays et al (2013) brings another type
of collaboration that can be interesting to measure at REBRATS, the collaboration in
the implementation / translation of evidence into decision making.
METHODS RESULTS
RESULTS
Eight studies were reviewed in search of data for: purpose of the
study, data collection instrument, collection strategy, collection period and
response rate. It should be noted that there were no studies focusing on health
technology assessment network, which highlights the pioneering spirit of Brazil in
this area.
Most studies used online data collection, with an email call. It was also held
collection by telephone and by letter, especially to encourage those who had not
yet responded via email. The data collection period was short varying from 2 to 6
months for most studies. During the data collection period, those who had not yet
responded also received an e-mail reminder. Another strategy that seems to have
been effective was the request to the directors of different centers to stimulate
other researchers of the centers to answer the questionnaires.

3. A tool to monitor and evaluate a HTA network: the case of REBRATS
Leão,L.S.C.1; Vanni,T.1
1 Department of Science and Technology, Brazilian Ministry of Health
Based on these data, an online survey strategy was developed. The call will
be done by e-mail to all members of the network. The form will be available to fill
out for a month. After this time, members who have not sent their response will be
recollected by email and phone and given a deadline of five working days. Data will
be collected annually.
As part of the implementation strategy, it is that the participation on the
survey is mandatory to all institutions members of REBRATS, and that it would
serve as a network member certification. It would be a sine qua non condition for
the permanence of the institution on the network on the following year.
For this strategy to be implemented, the inclusion of this new obligation on the
internal network regiment is required. This proposal must be submitted and
approved by the Executive Committee of REBRATS.
Data Analysis
Socioeconomic data will be analyzed using STATA / IC 12.1 for Windows
(StataCorp LP, Texas, USA). Confidence intervals for discrete variables ratios will be
calculated using the logit transformation, so that the end points are situated
between 0 and 1. Common confidence intervals will be used to mean the
continuous variables. The geographical origin and distribution of participants will
be plotted using ArcGIS Desktop 10 (ESRI, Redlands, USA). The social network data
will be organized using a program developed in C ++ to reorganize the data in a
mixing matrix. This data is then analyzed in Gephi, which is an open-source
network analysis software for viewing and operation of networks and complex
systems.
CONCLUSIONS
Survey Development
Based on the information collected in the eight selected studies and also based on
the specific characteristics of REBRATS, the survey called "Annual Report of the
activities of the REBRATS member institutions" was elaborated. The survey has the
following sections:
1. General Information
Institution’s name, address, e-mail, phone, expertise, kind of studies developed (ex.
systematic review). Team contacts and curriculum.
2. Studies Production and Dissemination
This session aims to quantify the production of studies and identify which were
conducted in collaboration with another network member institution. It also has
specific questions about the use of the study, as “This study has been used to
inform a decision making? What kind of decision?”
3. Participation at network activities
In this section the institution must give details of its participation at network
activities during the year, such as working groups meetings, events and trainings. 4.
4. Activities Promoted by the Institution
In this session the institution should report the HTA activities developed during the
year, like trainings, workshops, etc. It should also inform if the activity was done
with the collaboration of another network member.
5. Final Comments
This section is free for general comments.
RESULTS RESULTS

4. A tool to monitor and evaluate a HTA network: the case of REBRATS.
Leão,L.S.C.1; Vanni,T.1
1 Department of Science and Technology, Brazilian Ministry of Health
REFERENCES
1. Scientific collaboration and team science: a social network analysis of the centers for population health and health disparities. Okamoto J; Centers for Population Health
and Health Disparities Evaluation Working Group.Transl Behav Med. 2015 Mar;5(1):12-23. doi: 10.1007/s13142-014-0280-1.2.
2. Research collaboration in the discovery, development, and delivery networks of a statewide cancer coalition. Provan KG, Leischow SJ, Keagy J, Nodora J. Eval Program
Plann. 2010 Nov;33(4):349-55. doi: 10.1016/j.evalprogplan.2009.12.005. Epub 2009 Dec 16.
3. Identifying emerging research collaborations and networks: method development. Dozier AM, Martina CA, O'Dell NL, Fogg TT, Lurie SJ, Rubinstein EP, Pearson TA. Eval
Health Prof. 2014 Mar;37(1):19-32. doi: 10.1177/0163278713501693. Epub 2013 Sep 8.
4. Implications of network structure on public health collaboratives. Retrum JH, Chapman CL, Varda DM. Health Educ Behav. 2013 Oct;40(1 Suppl):13S-23S. doi:
10.1177/1090198113492759.
5. Co-evolution of departmental research collaboration and scholarly outcomes.
6. Katerndahl D. J Eval Clin Pract. 2012 Dec;18(6):1241-7. doi: 10.1111/j.1365-2753.2012.01881.x. Epub 2012 Jul 9.
7. Engaging Public Health Settings in Research Implementation and Translation Activities: Evidence from Practice-Based Research Networks. Mays GP, Hogg RA, Castellanos-
Cruz DM, Hoover AG, Fowler LC. Am J Prev Med. 2013 Dec;45(6):752-62. doi: 10.1016/j.amepre.2013.08.011.
8. Leadership in complex networks: the importance of network position and strategic action in a translational cancer research network. Long JC, Cunningham FC, Wiley J,
Carswell P, Braithwaite J. Implement Sci. 2013 Oct 11;8:122. doi: 10.1186/1748-5908-8-122.
9. A comparative study of 11 local health department organizational networks. Merrill J, Keeling JW, Carley KM. J Public Health Manag Pract. 2010 Nov-Dec;16(6):564-76.
doi: 10.1097/PHH.0b013e3181e31cee.
10. Newman MEJ, Barabási A-L, Watts DJ (2006) The Structure and Dynamics of Networks: Princeton University Press.
11. De Nooy W, Mrvar A, Batageli V (2011) Exploratory Social Network Analysis with Pajek: Cambridge University Press.
12. Easley D, Kleinberg J (2010) Networks, Crowds, and Markets: Reasoning About a Highly Connected World: Cambridge University Press.
13. Brandes U (2001) A faster algorithm for betweenness centrality. The Journal of Mathematical Sociology 25: 163-177.
14. Page L, Brin S, Motwani R, Winograd T (1998) The PageRank Citation Ranking: Bringing Order to the Web.