A Fun Study on 995 UIST Papers From 1988 to 2015
This post was written from work done with my advisor Philip Guo.
Motivation
During the winter break of 2016 I did a wonderful trip from Boston to New York City. After I came back to Rochester, I did another mental trip inside a bunch of HCI papers, because Philip and I wanted to do an investigation on the technologies used to implement HCI prototype systems. We chose UIST (User Interface Software and Technology) as our data source because it’s more system related 1. Since we were only interested in the implementation technologies, I didn’t do a full reading on each paper (unless they were really interesting to me). Instead my procedure was to scan over a paper, searched for the key sections talking about the implementation details, and then key words for any technologies (programming languages, toolkits, framework/libraries, platforms, etc) used in the implementation. I started pretty slowly, trying to go over each paper word by word, but then as I read more I learned (human learning!) to find certain patterns and heuristics to get the key words I need. It still took me a few days to read and analyze the 28 years of UIST papers though. All the data were recorded in a Google spreadsheet.
1 It might make more sense if we can analyze all the submitted papers. @bjo3rn
We then analyzed the data and visualized the data using d3, to answer the following questions:
What Kinds of Systems Are Presented Each Year?
The following figure shows a growing prevalence of hardware papers in UIST:
We can see that the UIST conference started as software-only, but in the past decade, nearly half of the papers each year had some hardware component (some were hybrid software/hardware systems). 2
In addition, the following table shows a progression from classic desktop GUI topics in the 1990s to the Web platform in the 2000s and then recently to mobile, multi-touch, and 3-D related topics that move away from personal computers. This shift in focus reflects the branching out of HCI systems (and theory) research beyond the classical desktop computing paradigm and also the decreasing cost of hardware prototyping toolkits (e.g., Arduino, Kinect, Oculus Rift).
| Era | Most common word | 2nd most | 3rd most |
|---|---|---|---|
| 1988 – 1991 | graphical | design | environment |
| 1992 – 1995 | information | graphical | speech |
| 1996 – 1999 | virtual | information | collaborative |
| 2000 – 2003 | input | text | mobile |
| 2004 – 2007 | web | displays | multi |
| 2008 – 2011 | touch | mobile | input |
| 2012 – 2015 | touch | 3D | sensing |
Note that hardware-based papers present more challenges for reproducibility. Even if those papers mention implementation technologies and link to detailed assembly instructions, it will still take specialized human expertise to be able to assemble, tune, and operate that hardware. In contrast, software is easier to set up and run, given good documentation.
2 It sometimes can be hard (and fuzzy) to precisely classify an HCI paper into the Software/Hardware/Joint categories. Different researchers might have different opinions even for the same paper. @bjo3rn @msbernst
How Many Papers Mentioned Their Implementations?
The following figure shows that the majority of software-related UIST papers did mention their implementation technologies.
We do not have sufficient expertise to assess the hardware papers, but we know that most UIST papers had some software component. In addition, 10% of all UIST papers (not shown in figures) were non-systems papers; those include UI design guidelines, meta-analyses, case studies, and opinion pieces.
We were curious about whether mentioning implementations in papers affected their citation counts, so we obtained the number of citations for all UIST papers from Google Scholar. However, we could not find statistically significant differences in distributions of citations for papers with and without implementation technology mentions (Mann-Whitney U test: U=48321, p=0.45). Papers mentioning implementation technologies had a mean of 72 citations (sd=105), and those that did not had a mean of 74 citations (sd=94). Similarly, there were no statistically significant differences in citation counts between software- and hardware-based papers.
Which Programming Languages Are The Most Popular?
The following figure shows the progression of programming languages used to implement software systems prototypes throughout all 28 years of UIST papers:
Traditional desktop languages such as Lisp, C, and C++ were the most popular until the mid-1990s. Then Java was invented in 1995 and quickly grew to dominate for the next decade on the desktop (and the incipient Web) due to its large standard library for GUI development and cross-platform deployment capabilities. In the most recent decade, the Web platform has dominated as browser technology has grown more mature, led by languages such as JavaScript for creating in-browser UIs coupled with Python and Ruby for server-side programming.
These historical trends align well with the TIOBE Index of language popularity in industry settings, except that Java and C still continue to dominate in industry today due to enterprise applications and low-level systems programming being far more prevalent than research prototypes. Given recent trends of HCI systems that combine hardware with software and that reach beyond the desktop, we predict that the next era of popular prototyping languages will be those that best support post-PC environments.
Tool Prototype: A Technology-Centered Paper Browser
While analyzing data for this post, we developed a set of Web-based visualizations to aid our own exploratory work. We believe that these visualizations can be extended to serve as a technology-centered paper browser: Instead of sifting through papers by titles or years, people should also be able to browse by implementation technologies. This kind of tool may help novice researchers or practitioners learn what technologies are available to implement their own related project ideas.
The above figure shows an excerpt from one of our prototypes, which is an interactive Sankey diagram in D3 that maps the relationships between programming languages, frameworks, and individual papers. Clicking on each paper title reveals its abstract and a link to its entry in the ACM Digital Library.
SUMMARY: Why Mention Implementations?
We have presented an overview of HCI system implementation technologies from all 28 years of UIST papers. But why even care about implementations in the first place?
First, implementation details contribute toward reproducibility of research systems. Ideally the source code for each system would be openly available, but that is not always feasible, especially in commercial research settings. However, even if code is not available, mentioning the programming languages, frameworks, libraries, toolkits, and other technologies used to implement the project can help future researchers re-implement their own versions to extend the project’s ideas.
Implementation details can also give students, hobbyists, and novice researchers a sense of what practical technologies are available to use in their own projects. Seeing what kinds of languages and frameworks are on the rise in recent years points to emerging technologies that may be good to learn, and conversely, seeing what is decreasing in popularity points to what may not be worth the time to learn in the near future.
Novices in particular get easily frustrated by the difficulties of implementing research systems since they are not adept at choosing the optimal abstractions. For instance, a novice might choose to use raw HTML5 WebSockets for bidirectional communication in a multi-user Web application, but raw WebSockets have many known quirks that lead to subtle bugs, especially across browsers. The Socket.IO library is a robust abstraction over WebSockets that makes it much easier to quickly create robust prototypes. Several UIST papers mention this library in their implementation sections, so a novice who reads those papers can learn about that library and save themselves a lot of time in their own implementation efforts.
Finally, since HCI research systems provide a glimpse into the future of technology, industry practitioners can study implementation trends to inform their own strategies for developing future products. Having a better sense of what researchers are using to implement their ideas can give companies a competitive advantage in rapidly prototyping new product lines.
In sum, we feel that every systems paper should mention their technology stack. It would be even better if they could list the technologies in a dedicated keywords section at the beginning of the paper and in indexable metadata. It takes very little effort on the part of the authors, and could have great benefits for researchers, students, and industry practitioners.