Interfaces and affordances

Material culture may be most significant, as Daniel Miller (2008, 279) says, because ‘much of what we are exists not through our consciousness or body, but as an exterior environment that habituates and prompts us’. While in the bustle of everyday life, objects may ‘fade out of focus,’ this serves only to ‘obscure their role’ and make them appear ‘inconsequential’, when they actually wield substantial power to shape people’s ‘behaviour and identity’. Miller adds that products have the nature of a ‘“distributed mind” which turns [people’s] agency into their effects, as influences upon the minds of others.’

Media can surely be seen in these terms. They have long been a regular feature of modern life, though their chief influence has usually been sought in the creative products that constitute their intangible ‘content.’ However, along with Miller’s stress on the neglected power of the object-world, Friedrich Kittler’s (1995) admonition that ‘there is no software’ asserts the possible primacy of media materiality. Kittler stresses that, ‘software does not exist as a machine-independent faculty.’ Which is to say, the forms of immaterial media content (music, movies, text) and therefore the experience of its consumption (imaginative stimulation, behavioural effects, informational enrichment) are strongly determined by the technologies that produce, distribute and present them for consumption. People encounter a medium’s content only in the terms of that medium’s technological materiality and through their relationship to both. Jason Farman (2012, 62–64) says this three-part formulation constitutes an interface, and that content-artifactuality-user relations in practice are inseparable. Here, for purposes of analysis, the focus is on the latter two. And the notion of interface is used more narrowly to refer to the device whose design affords quite specific relations with a given medium (and of course its content).

The concept of affordance is relevant here. Stig Hjarvard (2008), for instance, uses it in a discussion of the mediatization process, which the paper returns to below. James Gibson, whose term it is (‘the word is not to be found in any dictionary,’ he admitted (Gibson 1977, 67), said that an affordance is what the environment ‘provides or furnishes’ (1977, 68), and that ‘a way of life is a set of affordances that are utilized’ (Gibson 1977, 69). Gibson believed that an affordance is ‘uniquely suited’ to a given animal, like a human being (Gibson 1977, 79), and further that ‘an invariant variable that is commensurate with the body of the observer’ (Gibson 1977, 82) will be more readily perceived (and perhaps used) than one that is not. The anthropologist Tim Ingold (1986, 2–3, 7) adopts affordance to label the possibilities inherent in the raw materials of the natural world. Crucially, socialized humans have internalized these possibilities, and recognize, select and organize the raw materials according to their project at hand. Donald Norman means something else. A cognitive psychologist and design expert, Norman put the term into wider circulation with his 1988 book, The Psychology of Everyday Things, later retitled The Design of Everyday Things (2002). Norman distinguishes between an affordance and a perceived affordance. The former, in the case of a computer, simply refers to the capabilities of the machine; they are built in and the affordance ‘exists independently of what is visible on the screen.’ Displays on the screen, the perceived affordances, ‘advertise the affordances’ within (Norman 1999, 40). For the computer maker, ‘affordances specify the range of possible activities, but affordances are of little use if they are not visible to the users’ (Norman 1999, 41). Norman says in the preface to the 2002 edition of his book, ‘A good designer makes sure that appropriate actions are perceptible and inappropriate ones invisible’ (Norman 2002, xii).¹

In Norman’s sense, then, affordance refers both to a media functionality (radio is a sound medium; it cannot display visual images) and how a medium’s design invites the use of these functionalities. Gibson and Ingold would note that the built environment, within which most modern social life occurs, is replete with material objects, including media that present themselves to persons. Individual subjectivity determines significantly what people engage with, but a given medium’s capacities – the range of its possible uses – exist independently of a person’s perception of the medium. The design of a medium, especially the ways in which its potential uses are made available to a potential user, not least in relation to the literal features of the human body, will be a decisive element in people’s adoption and use of the device.

Interface relations

In much of her work that explores relations between people and information machines, Sherry Turkle invokes Donald Winnicott’s object relations theory. Winnicott’s interest was in child development. He observed that infants form bonds with objects, which for him include humans, not least the mother, that are crucial to the child’s understanding of what is me and not-me and to establishing trust that the child’s assertion of independence involves the certainty that the object will remain reliably available when needed again. Winnicott said that the relationship itself was distinct from the child and any particular object. If successful, the relationship becomes a model for life, a fruitful psychologically transitional space where adult fantasy, play and creativity all can flourish. Winnicott’s ideas have informed Turkle’s systematic analyses of the increasingly emotional relations people have with new media (see J. Miller 2014a, 112–113). This includes her meditation on scientists’ remembered relations with treasured childhood objects – their ‘falling in love with things’ (Turkle 2008, 38) – and their significance for occupational choice and even perhaps the person’s discipline and style of scientific investigation. Turkle’s (2008, 20) chief point is that, ‘What remains is a special way of experiencing objects that recalls this early experience of deep connection. Later in life, moments of creativity during which one feels at one with the universe will draw their power from the experience of the transitional object.’ Apart from Turkle, few media researchers have drawn on this theory. Roger Silverstone (1993) is one, but his focus is on how television’s formulaic content and ritualized consumption practices contribute to feelings of personal security. A more recent example applies it astutely to consumer behaviour. Ian Woodward (2011, 380) observes that, ‘all engagements with objects are creative … always constructive in one way or another.’ And, following Winnicott, that the transitional space is one of ‘experimentation, play and imaginative action where pragmatics and imagination must work in unison. Within it, desires are materially engaged and from it, new lacks emerge.’

Object relations theory suggests that people’s engagement with the affordances designed into the physicality of media devices means far more than mere instrumental usage – a measure of user friendliness, say. Rather, people are likely to approach media by deploying lifelong orientations to things. These may include an openness to find pleasure, companionship or utility in them. They may in this relationship draw the external object into their selves, experiencing the inanimate object as being alive. Or they may feel themselves to be part of the machine, taking its perspective on the world (Alexander 2008, 6–7). In these ways, one seemingly enters the medium ‘through the interface’ (Bodker 1990). Perhaps the more natural its design, the better it might function as a transitional object, a relationship that fosters intensely felt emotion. That is, the less machine-like a media device, the less explicit the tacit knowledge required to operate it, the more that natural activity like speech, touch and gesture cause it to respond to the wishes of the media user, the richer and the more intimate the connection between media user and medium. And, to return to Woodward, the more desire is intensified for later and more regular engagements in the quest to banish the inevitable sense of incompleteness.

Dematerialization of the interface

A more natural user interface (NUI) design will invite media access without the human user having to conform so much to the requirements of media technology – mouse and keyboard, special physical placement, etc. This can be glimpsed today in products like Microsoft’s Kinect and other video game motion-controllers. But dematerialization also and perhaps chiefly implies the distribution of media functionalities. Today’s increasingly wireless environments, from home audio to GPS mapping and social media on the run, are a foretaste of what is to come. Commonplace mobile media like the smartphone and tablet are highly portable and deeply personal objects, with touch screens and natural language capabilities that minimize a sense of artificiality in their use (J. Miller 2014b). Wearable wristbands and jewellery are primary means of monitoring, recording and transmitting one’s vital signs, whether for the prosaic care of chronic medical problems or the self-tracking activities of the quantified-self movement (Viseu and Suchman 2010, Ruckenstein 2014). Radical change will come in the next steps of interface transformation.

The likelihood of an internet of things (IOT) rests on the development of ever smaller and cheaper computers and sensors and their interconnection capability. Some of these devices are nano-scale (Akyildiz and Jornet, 2010). The market research firm IDC (2014) defines the IOT as ‘a network of networks of uniquely identifiable endpoints (or “things”) that communicate without human interaction using IP connectivity – be it “locally” or globally’, and foresees a $7.1 trillion worldwide market for it in 2020. Predictions are for tens of billions of things to be connected by that year. Wired magazine labels these conditions ‘the programmable world’ and offers an example of one house in which more than 200 objects are already connected (Wasik 2013; see also Chui et al 2010). Current examples extend to ‘pop-up’ bus service, like Bridj in Boston. Riders’ locations and needs entered online determine bus routes and schedules, which ‘dynamically deploy a transit network’ of nonstop transportation (bridj.com, Seelye 2014). The Bridj model calls for increased data collection and analysis to allow real-time anticipation of travel needs related to special events and the proper sizing of buses (Johnston 2014). The IOT has become a regular subject of academic and commercial conferences. The June 2014 MIT Technology Review Digital Summit in San Francisco, for instance, whose theme was the ‘connected world,’ involved participants from numerous industries and major universities and was covered by news organizations like the Wall Street Journal, New York Times and Bloomberg (see MIT Technology Review Business Report (2014).

In an internet of things, what (and where) is a medium in the conventional sense of the term will be increasingly hard to determine. This will be further complicated by the increasing intelligence of objects and environments, not only by their connectedness or embeddedness. Two related developments that undergird smart things and spaces are machine learning and genetic programming, or evolutionary computation. Both are concerned with building devices capable of autonomously improving performance based on experience. Machine learning designs learning algorithms that find their application in a variety of present-day tasks, including speech recognition, computer vision, bio-surveillance (tracking patterns of disease outbreaks), robot control and data mining. Machine learning is one way to design software when it is too difficult for humans to do. It is also a means to customize existing software to conform to a specific use (Mitchell 2006). Genetic programming adapts Darwinian principles of natural selection to allow software, over several generations, to evolve itself to best fit the usage at hand. John Koza (2008, 185) observes that it already, ‘routinely delivers human-competitive machine intelligence for problems of automated design and can serve as an automated invention machine,’ and offers numerous examples to make his point.²

Surveying these sorts of general developments, Brian Arthur (2009, 203, 207) takes the view that new technologies ‘become potential building blocks for the construction of further new technologies. The result is a form of evolution, combinatorial evolution …’ He perceptively adds – without reference to the IOT - that, ‘technology is no longer a machine with fixed architecture carrying out a fixed function. It is a system, a network of functionalities – a metabolism of things-executing things – that can sense its environment and reconfigure its actions to execute appropriately’.

Media in the car

The experience of the automobile as a mobile zone of privacy, safety and pleasure that moves through public spaces with the potential for danger and boredom has a long association with in-car media. A 1930 magazine ad for a Philco car radio offered this enticement: ‘Learn the thrill of having music with your mileage – the charm of riding to entertainment – getting everything that’s going on – missing nothing. You’re never alone with a Transitone’ (Matteson 1987, 75).

In these early years, long before multiple-vehicle households were common, listening to the car radio was rarely a solo activity, but very much an extension of domestic life.³ Mid-thirties research by CBS and NBC discovered that fewer than one in ten people did so alone (Russo 2010, 172). Analysts frequently say that in-car media create a ‘cocooning’ effect. Bijsterveld and her colleagues (2014, 7) use the term ‘acoustic shielding’, noting that while the car radio protects the driver from falling asleep and ‘developing a bad temper in heavy traffic’, it also uses sound production to mask the outer world. Michael Bull’s (2001, 364) research indicates that today the automobile’s ‘auditized space’ may be one of few opportunities for people ‘to do nothing without having to appear to be doing something else,’ and so has now become prized as a solitary place. Bull (2001, 371) views ‘automobile habitation’ as offering a unique ‘sanctuary’ that is ‘enhanced through privatized listening’.

According to Justin Williams (2014, 110; see also Matteson 1987), ‘Early anecdotal evidence suggests that car audio experimentation occurred soon after the turn of the 20^th century’, with Chrysler offering a factory-installed radio in 1922. Motorola (‘motor’ + ‘victrola’) built the first successful AM radio around 1930. It cost the equivalent of $1650 when an inexpensive car could be purchased for the equivalent of $9000. And the radio’s size was large, as much as two feet wide, eight inches high and 16 inches deep (Cortez 1996). The names of car radios were revealing. Crosley made a model called the Roamio and RCA later produced the Magic Brain radio, ‘free from crackle, spark and sputter – a new world of radio pleasure’ (Matteson 1987, 149). Hobbyist magazines offered instructions for DIY car radios. By the end of 1932, some 60 manufacturers were making automobile radios in the US (Matteson 1987, 112). As the radio’s physical size was reduced, designers chose to make it the stylish centerpiece of the dashboard. Williams (2014, 110) reports that the price dropped rapidly, and that in 1935 one million car radios were sold. Features like preset pushbutton tuning and foot controls appeared later in the 1930s. During the 1950s, when radios that could be removed and used as portables, car phonographs and transistor radios were all introduced, sales rose to five million annually (Cortez 1996); about 7.9 million cars were produced in 1955 (Walsh 2004). Williams (2014, 112) says that while in 1952 about half of American cars had radios, by 1980, ‘the start of a decade which saw the rapid growth of both the car audio aftermarket and hip-hop music’, the number grew to 95 per cent.

Factory-installed FM radios began appearing in American cars during the 1970s. Around the same time, Motorola offered eight-track tape players, which were soon replaced by smaller, more easily used audio cassettes. Citizens band (CB) radio, a kind of wireless point-to-point communication that others could hear, enjoyed a period of intense, somewhat counter-cultural popularity, especially among long-haul truck drivers. Compact discs featured digital recording quality, and cars in the 1990s had multiple-disc changers. Around the turn of the century, the combination of better sound insulation and improved sound reproduction technology turned cars into a ‘concert hall on wheels’ (Bijsterveld et al. 2014, 170). Subscription-based satellite radio now comes installed in the majority of American new cars. Sales of in-car consumer electronics in 2007 amounted to $10 billion (Williams 2014, 114).

The steady occupation of the automobile by infotainment technology is an instance of mediatization, which labels, as Sonia Livingstone (2009, ix) puts it, an ‘environment characterized by diverse, intersecting and still-evolving forms of multimodal, interactive, networked forms of communication’ that are ‘digitally convergent, hybridized, remediated and intertextual’ (see also Couldry and Hepp 2013, Hepp 2013, Hjarvard 2013). As ownership of cars steadily became commonplace during the pre-World War II period in the US, so too were automobiles increasingly equipped with AM radios that grew smaller, less costly, more integrated and produced better sound. The post-war era saw the rapid adoption of FM radio, better audio speakers and various formats for playing recorded music in the car. Both sets of developments mirrored such changes in media production, distribution and consumption as the dominance of commercial network radio, the LP record, high fidelity audio in the home, 1960s youth culture, alternative radio and hip-hop – along with the wholesale ‘automobilization’ of society (Thynell 2001, 59–60). For manufacturers and drivers alike, automobile travel became a unique opportunity for audio-enhanced, personalized solitude in a mobile space whose media-centric design increasingly masked its essential mechanical-transport nature. Having become a mediatized phenomenon, the car would next, and even more rapidly, become a site of what might be called digitalization.

Digitalizing the automobile

The digitalization of the car moves it beyond being a site of media consumption, albeit one with an unusually long history and dramatic cultural resonance. Digital media are now fundamental to the very conception, design, manufacture, operation and experience of the automobile. The occupants of a car are not only surrounded by audio and video media; the car itself is in constant networked interaction with the larger world for both its own operating reasons and for affording pleasure and useful information to driver and passengers. The human-digital media interface is multiple: tactile controls on the steering wheel, on the dashboard, between the seats; voice controls; wireless and wired exchanges between the car and personal media. At its extreme, the fully digitalized automobile becomes self-driving, utterly automated, leaving, in theory at least, its human occupants freedom to immerse themselves in on-board media.

A car today contains between 20 and 50 electronic control units (ECUs), or ‘computers’, depending on the car’s size. They constitute a system so complex that the wiring harness, which connects the ECUs by means of a controller area network (CAN), is one of the car’s heaviest components. Numerous sensors report information about tire pressure, fluid level and temperature, speed and so on. There is hardly a domain of automobile operation that is not digitalized. Cars continuously monitor themselves, making adjustments in response to changing conditions and conducting self-diagnoses that can be interpreted by a mechanic’s computer when repair is required. ECUs oversee engine management with respect to emissions and performance. Mechanical linkages have been replaced by drive-by-wire arrangements, such as electronic throttle controls or, in the case of hybrid cars, partially electronic braking. Automatic transmissions typically shift gears in response to electronic signals. Anti-lock braking systems (ABS) determine proper brake pressure to reduce skidding and promote optimum stopping distance. Automatic brakes sense when a car is too close to the car in front of it and reduce speed. Sometimes this is done independently of the driver’s actions. Climate control systems cool, heat and dehumidify a car’s interior, and wiper blades clear rain and snow from the windshield, both automatically responding to sensory inputs. Car doors lock and unlock and an alarm is set or deactivated with the use of keyless radio transmission. Most of these and similar activities have become so discreetly embedded and natural-seeming that today the driver scarcely takes notice.

Digital automotive enhancements continue to be introduced. The driver’s view to the rear is improved by a rear-facing video camera. Selecting a driving mode adjusts throttle response, steering assist, damper (or ride) firmness and transmission shift points. Coupling a forward-facing camera and radar-based cruise control, lane-keeping assist determines if a car is drifting outside the lines painted on roads and automatically adjusts the electronic power steering accordingly. Night-vision systems display difficult to see objects that emanate heat, like animals and people. Experimental pedestrian-recognition systems visually scan the car’s environment, reading the images for human forms night or day. ‘Intelligent drive’ systems incorporate these and other features such as parking assistance, adaptive high beam headlights and collision avoidance. One experiment finds a parking spot for a driver just setting out on his journey, incorporating his driving behaviour to calculate arrival time. Voice-activated technology permits the driver to make and receive phone calls and adjust climate control, while being read aloud incoming text messages. Wireless connectivity to a smartphone can play music stored on the phone. Apple’s CarPlay system is built around the iPhone, and is or will be available from some 20 automotive marques, ranging from Ferrari to Suzuki. Google’s competing system is the Open Automotive Alliance, and uses the Android operating system. Audi, General Motors, Honda and Hyundai are its founding members.

Nearly all American cars come equipped with an event data recorder (EDR), or black box, which the US National Highway Traffic Safety Administration has proposed making mandatory. Its purpose is said to be improved auto safety through better understanding of accidents. This small device automatically records a variety of data, including a car’s speed, throttle position, steering angle, braking, seatbelt wearing and, in the case of an accident, impact speed and airbag deployment. A similar, optional device transmits data in real time, increasingly through smartphones, to auto insurance companies that offer discounts to safe drivers (use-based insurance, or UBI).

Auto manufacturers commit substantial resources to the continuing digital transformation of the car. The Volkswagen Group of America, for example, has operated an Electronics Research Laboratory (ERL) in Silicon Valley since 1998 that employs about eighty people, including software, electrical and mechanical engineers, human factors researchers and designers. Perhaps the most dramatic step in these efforts to digitalize the automobile is the autonomous car, whose research has been sponsored in part by the Defense Advanced Research Projects Agency (DARPA), the US government agency that oversaw the invention of the internet.⁴ Google has been a prime mover and is building 100 examples of its design. Its current version completely eliminates even the possibility of human intervention – there is no steering wheel, brake or accelerator pedal, a truly driverless car (Markoff 2014).