BootStrapping Databases: Repositorium And TripDroid

1. Introduction
   1. Motivations

      Whereas the development of the digital media has enpowered people with the ability to create and distribute documents of very high quality, still many of those documents are not shared, even when there is demand for it (This is known as the pareto law of web 2.0: 80% of the documents accessed are produced by a minority of 20% of the people).

      We assert that this lack of use of sharing tools is due in large part to the lack incentives for sharing, and we focus on the cases where such documents are already produced, when privacy issues are inexistant, and where the effort to share is minimal. A natural solution, taken by existing communities (e.g. repository of pedagogical problems in theoretical computer science of the Institute of Science and Technology of Japan), is the creation of collective repositories, which content accessed is conditioned to the participation to the community, yet this leaves open the problem of assessing this participation in a way which scales with its participation. A centralized evaluation, by a finite group of experts, will necessarily result in a limit in the growth rate of the repository.

      We propose an extension of this natural solution, where users of a collective repository participate by both contributing new documents and by evaluating new submissions, the quality of their evaluation being monitored by mixing a small amount of "challenges" among their evaluation tasks, in a way inspired by "ReCaptcha" word recognition challenges.

   2. Illustrating examples
      1. Structured Touristic information produced by tourists

         A tourist finding an error in a Tourism guide possess information which could be usefull to other tourists. Ideally, he/she would send an errata to the editors of the tourism guide (paper or digital) in order to help future tourists, yet there is no strong incentive to do so, apart maybe having your name in a future edition.

         Conditioning the access to information provided and checked by previous users to the participation to the guide, either by providing new information or validating information entered by others, creates such an incentive.

      2. Pedagogical Material produced by instructors

         The average lecturer at an average university produces lecture notes, slides, assignments, examens, marking schemes, solutions to the problems, which are highly reusable by his colleagues in other universities. Some do publish the best fragments of their production in books, yet the competition and large effort involved prevent most to do so, and results in the hardly desirable situation of expensive printed textbooks, and very few databases of digital pedagogical material.

      3. Sharing of Digital data and code in Research

         The Scientific Method requires by definition the replication of scientific experiments by distinct research teams in order to independantly confirm the results. While in theory peer-reviewed publications in journal requires a description of the experimental setup sufficient to allow the replication of the experiment, in practice essential elements must be shared by other means: laboratory of biological research use courriers to send samples across the world.

         In the cases where the essential elements of the experience are digital (such as software in computer science, but also survey data in social sciences, detailed experimental data in hard sciences, etc…), collaborative repositories should insure their diffusion, yet have failed so far [Olivier Temam, head of architecture group at LRI, University Paris XI, France, personal communication], the only strategy showing some results being competitions where participants have to share their data in order to register (centralized quality control).

      4. Other examples
         • Release of Anonymized Medical Data?
         • Source Code intensively commented and documented
         • sharing and validation of scientific publications
2. Prototypes
   1. Prototype Repositorium
      • Target:
        • Sharing and evaluation of digital documents, and in particular pedagogical material produced by instructors for their own usage.
      • Example of Usage:
        1. Arturo is professor of physics and has invested some time to write a set of documents (e.g. multichoice conceptual questions) for his course Physics101. He is interested in getting feedback and additions on it.
           • He initiates http://physics.repositorium.cl with
             • his documents A,B,C,D,E for sharing,
             • the simple quality criteria "Is this question correct and adequate for teaching Physics101?"
             • some examples x,y of documents which do not satisfy this criteria.
           • He points out his new repository to some colleages at other universities, among which Brigit and Carlos.
        2. Brigit connects to the repository as user and
           • evaluates the quality criteria on {A,x,y,C}, and wins some credit as she answered correctly according to Arturo's evaluation.
           • The repository records that her evaluation agrees with the official evaluation of the documents.
           • Brigit then spends some of her credit
             • to search and download documents from the repository, and
             • to upload her own documents F,G,z,w in the hope to receive credit for them in the future.
        3. Carlos connects to the repository as user and
           • evaluates the quality criteria on {y,D,F,z}.
           • Carlos wins some credit as he answered correctly according to Arturo's evaluation of y and D, and
           • the repository records that
             • his evaluation of y and D agrees with their official evaluation, and that
             • he accepted F and rejected z: those statistics will be added to the evaluations by other users.
           • Carlos then spends some of his credit to search and download documents from the repository.
        4. Arturo connects to the repository as expert and
           • sees that 4 unvalidated documents have been added to the 5 validated documents initially in the repository, among which 2 have been (partially) evaluated. Without waiting for more evaluations, he
           • checks and confirm Carlos' evaluations of documents F and z: those are added to A,B,C,D,E and x,y among the validated documents used to assess user's expertise.
           • Arturo does not consider documents G and w: he will wait for more users to first evaluate it.
           • Arturo happily constats that user evaluations agree with the one in the repository, otherwise he might have to inquire for a misclasification.
           • Based on the documents which were previously rejected, Arturo adds a new quality criteria, "Is this document free of English mistakes?". He validates a positive (A) and a negative (x) example for this criteria. All other documents in the database are now unvalidated for this new criteria, to be evaluated by the users before being validated by an expert.
      • Known (Most General) Issues:
        1. Textual Documents: The current version is limited to text documents. Future version will manage the inclusion of images, the display of latex and HTML code, and the management of pdf documents.
        2. Dynamic Repositories: The current version manages a single repository with a single set of quality criteria. The next version will allow any user to create a new repository with its own quality criterias and content.
        3. Detection of Duplicates: The current version does not detect the submission of documents already present in the repository, whether in the exact same form (which could be automatically detected) or in an altered form (which requires a human to assess the identity). A future version will add automatic detection and human identity challenges (triggered by identity of tagging, and serving to refine the tags).
      • Open Issues:
        1. Tags versus Clusters: A document might be of interest to more than one community: having it assigned to a single repository limits its usage, having it assigned to several repositories and evaluating it separately wastes resources. On the other hand, having one single repository for all documents supposes
           • one single set of quality criteria for all documents
           • some additional system to decide which user is able to evaluate which criteria on which document.
        2. The expertise model is flat: each user is expected to correctly evaluate documents among each quality criteria, and each expert is expected to validate any document among any quality criteria. If this is more or less ok in the case of instructors teaching a similar course, expected to be experts on all parts of the course; such a feature unfortunately limits the use of a collective repository for sharing between users of distinct levels, such as:
           • users experts in distinct fields learning from each other when a user A is expert in field 1 (e.g. Spanish) and learner in field 2 (e.g. Chinese), while a second user B is expert in field 2 while being learner in field 1;
           • users at distinct levels of expertise when users experts on a topic 1 (e.g. basic data-structures) and learners on a topic 2 (e.g. advanced data structures) exchange access to level 2 material by evaluating submissions by users learning level 1 material.
   2. Prototype TripDroid
      • Targets:
        • Foreigners (or long term tourists) who wish to share and update touristic information about the city visited on their smartphone.
        • Inhabitants of a city who wish to share and update the location of available resources (ATMs, Restaurants, Gaz Stations) which change over time.
      • Example of Usage:
        1. Arturo launches the application "Trip Droid" on his smartphone while visiting the "Place of Arms" of Santiago.
           • He sees the tags "place", "statue", "artists" associated to the location, and
           • submits the tag "water display", to be added to the database as unvalidated information.
        2. A bit later, Beatriz also visits the "Place of Arms" of Santiago and launches the application "Trip Droid".
           • she is new to the application and does not have any credit: she sees only basic information, and a suggestion to answer some evaluation query.
           • She solicits a random evaluation query about the location, and receives "water display", Arturo's submission, with the option to validate, invalidate, or pass:
           • she can clearly see the sources, so she validates the tag "water display" and add a tag "Post Office".
           • she won some credit and can see more information now.
        3. At the end of the month Charlie, working for the tourism office, connects to the application as administrator.
           • He quickly validates the tags submitted which have been evaluated by tourists such as Arturo and Beatriz in the same way more than 20 times, and
           • he checks the few tags on which contradictory validations have been received.
           • He downloads the information to the server of the ministry of tourism, to be distributed to tourism agency and offices across the city.
      • Issues:
        • Probabilistic Quality Control: Asking only one validation query for each interaction (preferable given the context of usage of a smartphone), one cannot mix validation queries with challenges in each interaction, and must resort to randomness, the probability of which depending on the expertise of the user. How the rules of update of this probability affect the truthfulness of the process is not clear.
        • Semi-hidden Expertise: Having only one validation query per interaction, the credit update after answering it potentially reveals information about which is a challenge of a validation query to the user. The current solution is to partially hide the result, showing only the rounded log of the user score.
3. Our work

   Our work so far has consisted in putting this concept in practice in the context of two applications:

   • TripDroid for georeferenced information, aimed at the collective production and checking of touristic information; and
   • Repositorium for digital documents, aimed at the collective production and checking of any type of pedagogical documents (limited to textual in the alpha version).

   Both projects are still in a cycle of development and user implementation. They have been developped so far by students under academic supervision, their source code being made publically available under the philosophy that was is paid by public money should be kept freely available to the public.

   Our future work will consist in

   • testing more extensively the alpha-versions of those projects with users (only TripDroid so far has been user-tested);
   • extend the functionalities of these prototypes (e.g. extension from textual to digital for both)
   • explore the mathematical concepts underlying the collective quality control system

   Collaborations, concerning both the development or the assessment of boot-strapping tools, are more than welcome.

4. Acknowledgement
   1. Development of the prototypes:
      • TripDroid
        • Jorge Romo,
        • Vanessa Peña and
        • Christian Palomares
      • Repositorium
        • Hernan Fierro,
        • Pablo Estefo,
        • Felipe Banados,
        • Nicolas Perez,
        • Mauricio Quezada, and
        • David Contreras.
   2. Comments on the general design:
      • Sergio Ochoa (University of Chile)
      • Barbara Poblete (University of Chile)
      • Jo Piquer (University of Chile)
      • Eric Mazur (Harvard University)
      • Therese Bield (University of Waterloo)
      • Hans De Sterck (University of Waterloo)
      • Julien Sebot (Intel)
      • Balint Miklos (Google)
      • Claudio Gutierrez (University of Chile)
      • Christo Dichev (Winston Salem State University, North Carolina)