Progress report: Assessment of E-Negotiation Users and Processes
January 2003 - December 2003

Team

Project 2 began in January 2002. At this stage the project is mainly concerned with empirical analysis of available negotiation data sets as well as the design and preparation of new experiments. The main data source is the database created using the Inspire ENS since 1996. In addition, experiments were performed with two student groups in Austria and Taiwan in March 2003 using two different ENSs, Inspire and SimpleNS. Furthermore, in collaboration with project 5, an e-negotiation tournament using six different ENSs was performed in November/December 2003, and data from this tournament will also be available for analysis shortly.

Impact of negotiators' characteristics on process, outcomes and ENS adoption

The Inspire database contains records of all negotiations carried out using the Inspire ENS since the system was made available on the Web in 1996. It currently contains records of over 2500 negotiations with participants from over 50 countries. Thus, it provides a unique opportunity to study negotiator behaviour in a multicultural context. However, in order to provide a consistent data base for analysis, the original questionnaires, which have been developed at early stages of the Inspire project to obtain some feedback from users, had to be retained. Furthermore, the user population represented in the Inspire database is rather heterogeneous, since utilization of Inspire has not been not a control variable and access to Inspire has been open to anyone. Hence, empirical analysis of the Inspire data is comparable to an explorative field study rather than the analysis of a carefully controlled experiment.

Since the environment represented by the Inspire negotiation records is rather different from the controlled and homogeneous setting in which user assessment studies are usually carried out, it is not possible to use established models from the area of information systems evaluation like the Technology Adoption Model (TAM). We therefore ad to develop our own Assessment Model for Internet-based Systems (AMIS). AMIS builds on key concepts from the TAM model, like usefulness, ease of use and intentions to use an information system, and provides the extensions and modifications necessary for the environment in which Inspire negotiations are carried out. Specifically, the following points are taken into account:

1. Since the user population of Inspire is heterogeneous, user characteristics like culture, gender and previous experience with NSS are explicitly considered as important explanatory variables.
2. Inspire u users experience a complete negotiation rather than performing only a few well-defined trial tasks. Therefore, actual system use during the negotiation is considered as an additional construct, and we consider experienced rather than perceived ease of use and usefulness.
3. We consider behavioural intentions not only to use Inspire, but also other ENSs.

The development and initial empirical studies of AMIS have been documented in several research reports and journal publications. Early empirical tests have led to two important results:

1. There is a strong influence of national culture of users on variables at the beginning of the causal chain (such as expectations and behavioural variables of the negotiation process, e.g. the use of textual messages), but this effect becomes diluted in later stages.
2. In their experiences and perceptions, users clearly distinguish between different features of a system. This is particularly the case for communication-oriented vs. analytical, decision-support oriented components of Inspire.

These two empirical phenomena have led to the formulation of Culture-Technology Link Model about the different impacts that various features of an ENS might have on negotiators from different cultures. The model contains four hypotheses:

1. The compatibility hypothesis,
2. The structure hypothesis,
3. The media bandwidth hypothesis, and
4. The identity hypotheses

These four hypotheses will form the basis of our future empirical studies, which will be carried out using Inspire as well as other ENSs.

Text analysis

Language-Action Perspective
The initial focus of this sub-project has been on the low-level properties of Inspire texts. We see it as one of the main concerns in this project to seek connections between our specific mid-range goals (understand, classify and organize the linguistic data, prepare a strategy of deeper meaning analyses) and the work of our partners in the e-negotiation project. To this end, we have focused on the parts of Dr. Schoop's research program related to natural language texts. We have studied her work in the framework of the Language-Action Perspective, and her work on document management

Computational Linguistic Perspective
This subproject began in May 2003. The first stage has focussed on a statistical elaboration of the text messages that optionally accompany Inspire negotiations. We have built a multi-faceted model of the language in those messages; we took into account the considerable amount of noise due to the varied linguistic competence of the users, mainly second-language English speakers. We have constructed a comprehensive syntactic and semantic lexicon of all words encountered in the Inspire messages, classified into several distinctive categories. The lexicon will assist us in testing a variety of hypotheses. They concern the relationship between words and phrases used in the Inspire messages and properties of the negotiation process and the negotiators.

The following summarizes the work performed till the end of 2003.

1. We have analyzed the Inspire collection and established a preliminary view of the scope of the problems to solve. The fields of required techniques and methods include Natural Language Processing (NLP), Machine Learning (ML), Statistics and Corpus Linguistics.
2. We have investigated the main challenges of the Inspire collection. Those we must deal with in this phase of work include the high amount of spelling and grammatical errors, the mixture of English and foreign words, the sparsity and irregularity of textual data.
3. We have conducted a (largely unsuccessful) bibliographical search for work in NLP that would deal with data similar to ours.
4. We have recognized the need to build new programming tools to perform the necessary NLP tasks. The tasks include lexicon construction, text pattern recognition, and statistical evaluation of text data.
5. We have carried out the lexical analysis of the Inspire corpus, and built a domain lexicon using available on-line dictionaries. We presented the results in the InterNeg Report 01/04.
6. We have applied statistical tools of corpus linguistics to the task of examining spelling noise and its relation to negotiation outcomes.
7. We have investigated several approaches to data representation that enable us to apply ML techniques. We ran ML experiments to find regularities in the Inspire textual data. Our preliminary results show that classification of negotiations by success based only on text data can be no less accurate than classification based on the negotiation process.
8. We have developed and implemented a procedure of finding text patterns indicative of successful negotiations.

Assessment of Negoisst negotiators and negotiations

So far, Negoisst has mainly been evaluated in real-life negotiations between members of cooperative construction teams. The negotiation tournament offered a different possibility of assessing the system's capabilities and shortcomings. Based on the pre-negotiation and post-negotiation questionnaire, we obtained information about the system's usefulness regarding the support of efficient negotiations.

Furthermore, we are also interested in the quality aspect of negotiation support since Negoisst is strongly based on communication management and dynamic support of user flexibility. To this end, the negotiation logs were examined and compared to the previous negotiations in the construction sector.

The main goal of the assessment and examination of negotiation steps is to improve and validate our approach to communicative support and enrichment of complex (written) negotiation processes conducted with Negoisst. The assessment is based on a framework of speech act elements that are taken, revised, and combined from Habermas' Theory of Communicative Action and Searle's Theory of Speech Acts. The framework has been successfully used in cooperative document management in healthcare and as a theoretical basis for the development of Negoisst. Each utterance is classified into one class of speech acts representing the sender's intention, e.g. assertive, commissives, directive. The particular illocutionary force (i.e. offer, request, information, question etc.) needs to be adapted to the field of negotiations. One of our aims is to refine the existing classification by evaluating whether the forces offered as message types so far are sufficient. Each class of acts then carries certain obligations. For example, a commissive represents a speaker's/sender's intention to carry out a particular action. Such obligations are important for the contract that is automatically derived from the messages that are exchanged.

Further research includes an assessment of potential application branches of Negoisst and continuous evaluations both within and beside the negotiation tournament. 

Comparative study of two ENSs use

While e-negotiations gain in popularity there have been few studies comparing the efficacy of different ENSs. We have initiated such a study with the comparison of negotiations performed with Inspire and the Simple NS systems. The novelty of this study requires both theoretical and empirical research. Both issues are addressed in ongoing work within this project.

The main theoretical issue arising out of this project is the ex post evaluation of negotiator performance. In NSS without analytical support, like SimpleNS, no utility functions are elicited from negotiators. Therefore it is not possible to compare negotiation results using utility values. To overcome this problem, the Extended Dominance Relation (EDR) method for the evaluation of negotiation outcomes has been developed. This approach starts from the dominance relation of multiattribute decision analysis, which provides an incomplete ranking of alternatives without requiring information about preferences of decision makers (negotiators).

In the EDR approach, the dominance relation is augmented by observed preferences between alternatives discussed during the negotiation. The method generates a scalar measure of negotiator performance, which does not require experimental subjects to estimate a utility function and thus can be used to evaluate outcomes achieved with any ENS. In a first empirical analysis using this approach, different variants of EDR performance measures have been successfully tested against utility ratings available from Inspire. Results of this study, as well as the theoretical development of the EDR approach, is documented in a working paper, and submission to a peer-reviewed journal is being prepared.

While the EDR approach can be used to measure the performance of individual negotiators, additional methods are required to determine the efficiency (Pareto-optimality) of negotiation results. For this problem, an extension of Data Envelopment Analysis as well as a probability-based approach were developed. Theoretical development and implementation of the software necessary to test these approaches have been completed and the initial results were presented at the project workshop in Prague. It is expected that this stream of research will lead to a journal submission in 2004.

For the empirical comparison of Inspire and SimpleNS, a set of controlled experiments has been conducted in March 2003 with 80 Austrian and Taiwanese participants using two ENS: SimpleNS and Inspire. Results of these experiments are being analyzed using statistical methods for quantitative characteristics of the users, negotiation processes and outcomes. In addition, content analysis methods are employed to obtain a deeper understanding of communication patterns and the negotiation processes.

Preliminary results of the quantitative analysis indicate that while there is little difference in the users' assessment of the SimpleNS and Inspire negotiations, a significantly greater number of Inspire users reached an agreement than the SimpleNS users. This would indicate that support based on analytical models does not have negative influences on the negotiation process but significantly impacts the negotiation effectiveness.

The content analysis performed on the negotiation logs of these experiments aims to describe and analyze web-based negotiation processes in more detail. In particular we want to:

1. Describe electronic negotiation processes and identify particularities of negation processes of each system.
2. Analyze the impact of the support system on negotiation behaviour.
3. Compare negotiation behaviour with regard to user characteristics (in particular gender), negotiation role, and culture.
4. Identify negotiation behaviour and/or patterns, which increase the probability of agreements.

We use qualitative research methods, i.e. content, process and interaction analysis to analyze the data. Currently, the negotiation process transcripts of 40 negotiations are separated in communication propositions, i.e. the unit of analysis, and coded by two independent coders. For this purpose, we have adapted the Bargaining Process Analysis (BPA) model for electronic negotiation processes. Additionally, we apply personal pronoun analysis. Preliminary results are expected in May 2004, to be followed by a research report and a conference presentation in July 2004.

Ph.D. and Master's theses:

Sokolova, M. (2003). "The language of negotiations." PhD, University of Ottawa (in progress).

Zaremba, R. (2003). MBA thesis, University of Ottawa, (in progress).

Journal publications

Vetschera, R., G. E. Kersten and S. Koeszegi (2004). "The Determinants of NSS Success: An Integrated Model and Its Evaluation." Journal of Organizational Computing, (submitted).

Koeszegi, S., R. Vetschera and G. E. Kersten (2004), "Cultural Influences on the Use and Perception of Internet-based NSS - An Exploratory Analysis." International Negotiations Journal, Vol. 9, No. 1, (forthcoming).

Kersten, G. E. and G. Zhang (2003). "Mining Inspire Data for the Determinants of Successful Internet Negotiations." Central European Journal of Operational Research, Vol. 11, No. 3, (297-316).

Kersten, G. E., S. Koeszegi and R. Vetschera (2003). "The Effects of Culture in Computer-Mediated Negotiations; Experiments in 10 Countries." Journal of Information Technology Theory and Application (JITTA), Vol. 5, No. 2, (1-28).

Yuan, Y., M. Head and M. Du (2003). "The Effects of Multimedia Communication on Web-based Negotiation." Group Decision and Negotiation, Vol. 12, (89-109).

Conference proceedings

M. Sokolova, S. Szpakowicz, V. Nastase (2004). "Using Language to Determine Success in Negotiations." Canadian AI, (submitted).

S. T. Koeszegi (2003). "Exploring the Culture-Technology Link. Does Culture Influence Behavior in E-negotiations?" EURO/INFORMS Conference, Istanbul , Turkey.

G. E. Kersten, S.T. Koeszegi and R. Vetschera (2003). "User Attitudes towards Web-based Intercultural Negotiations: An Extended TAM Model." EURO/INFORMS Conference, Istanbul , Turkey.

Technical reports

M. Jarmasz, S. Szpakowicz (2003). "A Study of the Language Used in Electronic Negotiations." Preliminary report. SITE, University of Ottawa.

M. Sokolova, S. Szpakowicz, V. Nastase (2004). "Automatically Building a Lexicon from Raw Noisy Data in a Closed Domain." Technical Report INR 01/04, (1-11).

S. Koeszegi (2003): "Exploring the Culture-Technology Link - Does Culture Influence Behavior in E-Negotiations?" Working paper OP 2003-08, Dept. of Business Studies, University of Vienna (35 pages).

P. Grünbacher, S. Koeszegi, M. Halling, S. Biffl (2003): "Integrating Negotiation Theory Concepts into Software Requirements Negotiations", Working Paper OP 2003-09, Dept. of Business Studies, University of Vienna, (33 pages).

R. Vetschera (2004): "Estimating Negotiator Performance Without Preference Information", Working Paper OP 2004-01, Dept. of Business Studies, University of Vienna, (24 pages).

Plan 2004-2006

Research will continue along the lines described above. For the empirical analysis of Inspire and further development of the AMIS model, the questionnaires currently in use in Inspire will be redesigned to more closely reflect constructs of the AMIS model. This change should lead to results which are more comparable with the existing literature on IS evaluation. The revised AMIS model will be tested using data obtained from the e-negotiation tournament performed in December 2003 as well as a new tournament, which will take place in January 2004.

Development of performance measures will also continue as outlined above. We expect a journal submission of the EDR method paper to be ready in the first months of 2004 and a working paper on the DEA and probability-based approach to be completed by mid 2004. Initial tests of the EDR method have identified a surprisingly high level of inconsistent behaviour by subjects: in more than a third of all negotiations analyzed, negotiators finally accepted a compromise, which, according to the utility values calculated by Inspire, was worse for them than a previous offer from their opponents. An empirical study of possible explanations for thus behaviour is planned for late 2004 and will continue throughout 2005. The performance measurement approaches developed will be employed to analyze empirical results obtained with different NSS throughout the project.

Work on text analysis will continue throughout 2004 and 2005. In the next several months, we will:

• Finalize the procedure to find patterns that indicate success of negotiations;
• Identify more semantic categories, use them to represent negotiations, apply classification algorithms;
• Relate technical and non-technical terms to noise, connect the relations to negotiation outcomes;
• Tag short N-grams and longer segments with semantic information;
• Parse the Inspire data and analyze results.

The longer-term plans include the following:

• Collect textual data used in other negotiation support systems;
• Link stages of the negotiation process with the textual data, investigate the dynamics of the negotiation process;
• Investigate the use of syntactic NLP tools (taggers, chunkers, parsers) for the task of analyzing the language of negotiations;
• Further investigate, using results of the parsing process, patterns indicative of the outcome of a negotiation;
• Conduct a preliminary investigation of the way in which the background and cultural influences of the negotiators are reflected in text messages.

Cooperation

Project members at the University of Vienna (R. Vetschera, S. Koeszegi) have started a cooperation with the departments of Software engineering at the Technical University of Vienna, Austria and the University of Linz, Austria to launch a joint research program on the use of decision and negotiation analysis in software engineering projects. A project proposal for funding of such a program has been submitted. Initial research in the preparation phase of this project has focused on identifying opportunities to use negotiation support in software requirements analysis (Grünbacher et al., 2003).

Planned Events

S. Koeszegi and R. Vetschera will organize the meeting of the INFORMS Section on Group Decision and Negotiation in Vienna 2005.