Etusivu › artikkeli › Artikkelit › 5. Delphi Manager

5. Delphi Manager

Kirjoittanut , • ( 0 )

Toni Stubin & Hannu Linturi (updated 29.2.2024)

A Delphi is initiated by a development need or a research problem, whose description determines the subsequent steps of the process. In the first phase, the task of the Delphi manager, or researcher, is to structure, delimit, and align the research questions together with the possible client of the study. The manager is responsible for defining the phenomenon to be studied or developed, which, if successful, energizes the entire research process. After that, there are two parallel tasks, which are examined in this article. One is the selection of the expert panel, and the other is the construction of the questions or future assertions from which the Delphi panel’s activity is initiated. The manager’s tasks, of course, continue after these, but those – process facilitation, data analysis, and reporting – will be covered in future blogs.

Assembling the Panel

Once the research or development question has been defined, the manager begins selecting the panel. The goal is to find participants whose expertise and stakeholder involvement cover the essential aspects of the phenomenon being studied. In a skillfully chosen panel, there is representation for all the plausible future options that can arise from the subject. In English-language methodological literature, the classification of the panel often uses the terms cognitive expertise (knowledge, understanding expertise) and social expertise (social, relational expertise). The manager must have at least moderate expertise in both to be able to select the correct expert composition and also to guide the panel’s activities during the research. The manager is like a conductor who must understand the possibilities of each instrument to successfully lead the orchestra. The panel is then the actual orchestra that produces the music.

The manager’s most important individual task is to map the expertise and stakeholder interests related to the phenomenon and select the panelists based on this. For example, in Delphi dissertations, the composition of the panel is invariably a central factor in the use and reliability of the method, and one should be prepared for challenging questions from opponents. If an essential expert group is missing or underrepresented in the panel, the reliability of the research may be questioned. When assembling the panel, it is worth spending a lot of time and thought creating a good and sufficiently diverse expert matrix. The selection of the panel is dealt with more closely in the previous blog post of the series.

The polyphony related to assembling the panel gives the manager the opportunity to strengthen the panel’s participation and interaction. Here, the aforementioned social expertise is needed. If the panel does not develop a lively discussion, argumentation, and dialogue, its results are likely to be meager. Well-assembled panels naturally develop tensions related to perspectives, knowledge, and interests. These tensions provide the manager with natural opportunities to communicate with panelists and materials.

The interaction facilitated by the manager in the panel may have features of both debate and dialogue, although the latter is actually what Delphi aims for. ’Being right and winning form the basic dynamics of debate, which is often stronger than the need to find the best solution. In a debate, the argument can be won or lost, but in dialogue, there are no winners or losers. In dialogue, the purpose is not to be right or wrong at all.’ (Holm, Poutanen, Ståhle 2018) From the definition, the original etymology of the word dialogue is still well recognized. The Greek prefix “Dia” means “through” and “logos” signifies “word” or “meaning”. In dialogue, meanings are formed through words. The Socratic method, which also inspired the evolution of the Delphi method, is an excellent example of this.

The manager can set four objectives to make the panel truly dialogical. In successful Delphi management,

  1. Panelists feel equal, which of course is ensured by anonymous participation.
  2. Dialogue is supported by emphasizing the importance of reading and ’listening’ to other panelists. A key indicator of listening is that panelists are able to reassess their own viewpoints.
  3. Delphi panelists are expected to be willing to examine and question their own background assumptions. Reviewing a panelist’s social competence (interest, stakeholder status) illuminates this aspect already when selecting panelists.
  4. Dialogue requires sufficient mutual trust in two respects: on the one hand, that the information received from others is valuable, and on the other hand, that participants can be trusted as people. If one of these dimensions is missing, the amount and quality of interaction will be limited. Building trust is the manager’s task.

Constructing the Query

Tutkittavan ilmiön systeeminen kuvaaminen ja rajaus johtaa ideaalitapauksessa suoraan kyselyn teemoihin ja auttaa tunnistamaan olennaisten kysymysten kohteita. Delfoin prosessiluonne mahdollistaa myös muut lähestymistavat kuin sen, että kyselyt ja kysymykset kattavat täydellisesti tutkittavan ilmiön. Se johtaa usein laajoihin ja panelisteja kuormittaviin kyselyihin, mikä saattaa vähentää laadullista ja  vuorovaikutteista tiedon muodostusta. Keskittymällä muutamiin kuumiin kysymyksiin tulos voi olla parempi. Silloinkin on tärkeää, että managerilla on hallussaan laaja ilmiökuva monine kysymyksineen. Niitä vasten hän voi havainnoida paneelin vastauksia ja analysoida panelistien kommentointia. Iteratiivinen prosessi mahdollistaa sen, että ilmiöta kuoritaan kuin sipulia yhä syvemmäkllä ja toisinaan myös laajemmalle. 

Monet hyvän kysymyksen tuntomerkit – kuten yksiselitteisyys ja ymmärrettävyys – ovat samoja Delfopissa kuin missä tahansa tutkimuksessa. Osa taas on tyypillisiä vain Delfoi-tutkimukselle. Tällaisia ominaisuuksia ovat esimerkiksi yllättävyys ja siihen sisältyvä radikaali muutos nykytilaan sekä kyky erotella vastaajia eri puolille asteikkoja. Kaikkien kysymysten ei tarvitse täyttää kaikkia mahdollisia kriteereitä, mutta paneelin käynnistymisen ja syttymisen kannalta kosketukset panelistien intellektuaaliseen ja emotionaaliseen persoonaan ovat myönteisiä. Delfoi-väite on kuin uutisotsikko, joka houkuttelee jatkamaan lukemista. Usein se haastaa olemaan puolesta ja vastaan. Siinä mielessä kysymys yllyttää debatoimaan olkoonkin, että sen perimmäinen tavoite on osallistujien välinen dialogi.

An essential part of constructing questions is finding the appropriate type of question for different situations. Popular types of questions are scale-based, but there are also specific areas of validity for, for example, time series, timeline, and categorization questions.

Before inviting panelists into the electronic panel environment, the manager must construct one or more future-oriented questions. When the questions are aimed at the future – where they have no truth value – they should be formulated in a way that allows panelists’ views to diverge without elevating any option above another. The study can easily be ’ruined’ at this stage by asking overly leading questions.

Good questions are open, clear, interesting, and divisive in terms of opinions. Such a question not only invites responses but also justifications for one’s own answer choices. Even though a question is set in the future, its roots are always in the present and current experience. In anticipation theory, the connection between times is expressed ingeniously. The problem or phenomenon is taken to the future for examination, so that it can be brought back to the present to be solved.

An essential part of question construction is finding the appropriate question type for different situations. Some questions differ from conventional forms. The eDelphi platform has over a dozen question types, which can be divided into four different groups according to their purpose.

The Basic Question category includes explanatory and contextualizing question elements. Among the options, the text field is actually a light editor, where you can edit text in various ways, as well as integrate media (image or video) and links into the document. Identical to Survey inquiries is the form, often used for gathering background information. The form can incorporate three types of questions: text, memo, or choice. A particular background question for Delphi is the expertise classification, where respondents themselves define the quality of their expertise. This does not replace the expert matrix made by the manager before the study begins but can provide interesting information on how the panelist perceives their own expertise and involvement.

Scale-based assertions are the most characteristic and common types of questions for Delphi. Here, the question thesis is addressed on a Likert scale, whose length and values are defined by the manager. Typical are, for example, five- or seven-point scales (strongly agree … strongly disagree). A question can include one or two criterion variables in different combinations. Criterion variables are an essential part of the scale question. They form the basis for evaluating the statement’s content. The most common evaluation variables are probability, desirability, significance, importance, and feasibility, of which the first two are the most popular combination. eDelphi has five types of scale questions, one of which is specifically designed to support voting, commenting, and analysis in Live and online situations.

Forecasting questions are typical future-oriented question types for Delphi. The most important of these is the time series question. It suits trend questions for which there is numerical historical data. In the time series question, the manager inputs historical data – often on an annual basis – as the basis for the question, and the panelist, in turn, estimates how the development will continue from there. The question shows previous development as a line chart, onto which the panelist plots the future development curve based on their choices. Time series questions are popular and neutral future questions for panelists, where it’s possible to take a stance up or down or even curvilinearly if there are three or more measuring points. The timeline question is often used in technology foresight. It asks panelists to estimate the timing of a phenomenon’s realization – such as the breakthrough of electric airplanes – on an annual basis. Respondents can also be given the option to define a time window in which the phenomenon occurs, i.e., mark the earliest and latest possible times.

Structuring questions are often used to orient and contextualize the panel to the phenomenon under examination. They all involve classifying a set of characteristics. In an ordering question, characteristics are classified according to a criterion into a consecutive sequence, for example, from most important to least important. In a grouping question, characteristics are placed into categories defined by the manager, such as rising, falling, and remaining the same. The multiple-choice question is a familiar type from survey research, where one or more options are selected from a list.

Many hallmarks of a good question – such as unambiguity and comprehensibility – are the same as in standard survey research. Some, however, are especially typical for Delphi research. Such qualities include interest, emotional engagement, surprise, and the ability to divide respondents’ views. Not all questions need to meet all possible criteria, but touching the panelists’ intellectual and emotional persona is positive for initiating and igniting the panel. Also, the dramaturgy of the survey is worth considering. It’s advisable to start with an interesting and easy-to-answer ’warm-up question’ that forces the expert to think about their answer but also allows quick progression. Questions on the same topic should be grouped together, and interest can be stimulated by placing provocative questions at regular intervals. The number of questions should be considered, as even motivated panelists’ attention wanes towards the end of long series of questions. Willingness to comment often diminishes the further the survey progresses – and then valuable qualitative information is lost. A good maximum length for one panel or panel round is about ten questions.

An absolute precondition for good dialogue is that respondents trust in the anonymity being maintained. Another type of trust is important for motivation, namely trust that one will gain inspiration, insight, and understanding from other panelists and the process. A panelist is best motivated when they feel that they also gain something from participating in the panel.

Formulating questions is not easy. Constructing questions rewards the persistent who are willing to refine questions with test respondents. An experienced researcher can often be distinguished from a novice by the fact that the latter does not yet know how to look beyond one round to the continuation process or the possibilities of using panel structure-related dramaturgy in the process.

A rule of thumb is that when formulating text, future assertions should be written in the indicative mood as if the events described in the assertion have actually occurred (e.g., ”By 2035, every third Finnish family will have a flying car”). Panelists should be reminded repeatedly that the assertion describes a certain future moment and the (imaginary) action or phenomenon occurring then. The panelist expert’s task is first to take a stand on the scale of the statement’s likelihood and desirability and then justify their choice in the comment box. Panelists should be encouraged to answer and justify their responses for each question – and of course to participate in discussion with others.

On the eDelphi platform, the manager has many ways to operate questions and surveys. Each question – and survey – is either visible or hidden. This feature allows the content of the Delphi round to be rhythmically planned according to a pre-scheme or situation. New questions can also be added to the panel as potential new future assertion topics emerge from the panel. There are other options for individual questions. Commenting can be specifically opened or closed.

The same can be done for the visibility of other responses, which has two different features. For each question, a view can be opened to the summative view (a diagram showing the distribution of panelists’ responses) and to all comments, which opens up the possibility for the panelist to also comment on other panelists’ comments. The creation of discussion threads dialogizes the Delphi process and is the reward for every Delphi manager of a successful process. Generally, it’s most profitable to enable commenting for each question and also to show all comments.

A common pitfall in creating questions is producing persuasive hidden meanings. Often the manager themselves represent the expert and interest groups of the phenomenon being examined, which can influence the choice and direction of the questions’ subject matter, or even the formulation of the question itself. Negative theses are avoided for good reason (respondents easily confuse the direction of the answer when asked negatively), as well as provocative statements leaning towards a certain interest direction. Provocative questions themselves need not be feared, as the respondent typically has at their disposal a scale with which they can relate the statement as they wish. A provocative thesis is apt to arouse interest and motivation. The future tolerance of panelists can be increased with pre-information, which opens up respondents’ minds to contemplation of alternatives and discussions with others. Questions can be supplemented with plenty of background information to support thinking.

Practical tasks include instructing how to respond to questions both before and during the survey round. At the appropriate stage, the manager opens the responses for real-time discussion in the panel. The panelist, in turn, can check or change their responses at any stage of the process, as well as attempt to influence the responses of other panelists. Responses and comments are anonymous at all stages, so that the respondent’s position or role does not influence the shaping of opinions. The manager also makes a choice – even before starting the research – whether to guide the panelists and the process towards consensus or to produce arguments for and against different futures. This depends a lot on the objective of the study: when drafting an action plan, consensus may be a good goal, whereas in creating scenarios, different arguments usually provide the best material.

Based on the results of the first round, the manager drafts the questionnaire for the second round, which deepens, expands, or focuses the discussion and argumentation of the previous round’s topic. Delphi rounds are typically conducted from two to four. Between rounds and naturally at the end of the research process, the manager analyzes and reports the research results and/or development suggestions, which in development projects often form the basis for decision-making.

Also possible is the so-called roundless or real-time Delphi, where respondents can enter the panel during its opening hours to respond, participate in discussions, and if necessary, change their responses, but new rounds are not organized. In real-time Delphi, the manager must maintain the panelists’ interest in returning to the survey also after their first response. This can be done, for example, by sending messages to the panelists, informing them about the progress of the panel and highlighting interesting observations, such as the distribution of responses so far and interesting arguments.

Delphi rounding and data analysis will be discussed in the following blog posts!

Literature

‹
›

Kategoriat:Artikkelit, blogi, Tie

Avainsanat:, , ,

Jätä kommentti

This site uses Akismet to reduce spam. Learn how your comment data is processed.