“If the future of banking is going to be digital, we want it to be populated with those who value the deeper tenets of open source philosophy. Otherwise we could be left with increasingly alienating, exclusive and unaccountable financial surveillance states, presiding over increasingly passive and patronised users.”
The following is excerpted from another excellent and illuminating essay from Brett Scott, about the dark side of the algorithms that are running much of work and life, behind the screens, and much of it already applied in the world of finance.
We recommend reading the original essay in full.
“‘Algo-robotic’ systems are particularly adept at accumulating power. Unlike the simple machine that offers static options via an interface, an algo-robot – or a series of linked algo-robots – have a greater ability to react in multiple ways in response to multiple data streams, and therefore to organise and co-ordinate. This trait makes senior corporate management warm to them, because, after all, reacting and co-ordinating are core elements of what a manager does.
The old hierarchy within a corporation was one where owners used managers to co-ordinate workers and machines. This gave rise to the traditional battles between owners and managers, managers and workers, and workers and machines. The emergent hierarchy is subtly different. The owners – often a disparate collection of distant shareholders – grant power to high-level management, who increasingly use algorithmic systems as ‘middle management’ to organise their workers and more basic machines.
And this is where we see the changing conception of the robotic system’s ‘body’. Rather than being a mechanical assemblage with an algorithmic ‘mind’, the robot could be an algorithmic mind co-ordinating a ‘body’ constituted out of ordinary employees, who increasingly act like machine parts. Think about the Amazon deliveryman driving the van to act out an order sent to him by an algorithm. This ‘body’ doesn’t even have to be constituted by the company’s own employees, as in the case of self-employed Uber drivers co-ordinated by the Uber algorithms.
These arrangements are often difficult to perceive, but algo-robotic systems have been embedding themselves into everyday forms of finance for decades, not necessarily ‘taking over control’ but often creating a hybrid structure in which manual human actions interact with automated machine-robot actions. For example, the investment bank trader might negotiate a derivatives deal over the phone and then book it into a partly automated back-office system.
The quintessential example, however, is the retail bank branch. You can talk with employees behind the Barclays counters, but often they are just there to enter data into a centralised system that tells them how to deal with you. To some degree these employees have agency – the ability to make quasi-autonomous decisions – but the dominant trend is for them to become subservient to the machinic system they work with, unable to operate outside the bounds set by their computer. Indeed, many bank employees cannot explain why the computers have made the decisions they have, and thus they appear as the human face put there to break the news of whatever the algorithm has decided. We might even say they are a human interface to an otherwise algo-robotic system that is accountable only to the senior corporate management, who you will never deal with.
But, ‘human interfaces’ like that are actually quite costly to maintain. People are alive, and thus need food, sick leave, maternity leave and education. They also have a troublesome awareness of exploitation and an unpredictable ability to disobey, defraud, make mistakes or go rogue. Thus, over the years corporate managers have tried to push the power balance in this hybrid model towards the machine side. In their ideal world, bank executives would get rid of as many manual human elements as possible and replace them with software systems moving binary code around on hard drives, a process they refer to as ‘digitisation’. Corporate management is fond of digitisation – and other forms of automation – because it is a force for scale, standardisation and efficiency – and in turn lowers costs, leading to enhanced profits.
The process is perhaps most advanced in the realm of electronic payments, where money is shifted with very little human action at all. Despite recent talk of the rise of digital currencies, most money in advanced economies is digital already, and tapping your contactless payment card sets in motion an elaborate automated system of hard-drive editing that ‘moves’ your money from one bank data-centre to another. This technology underpins talk of a future ‘cashless society’. Bouncy startups like Venmo and iZettle have got into the payments game, adding friendly new layers to an underlying digital payments infrastructure that is nonetheless still dominated by the banking industry and credit card networks.
In the case of retail banking, an ideal situation for banks might be to get rid of the branches altogether, and to push for a world of ‘branchless digital banking’. This generally means slowly dismantling, delegitimising and denaturalising branches in the public imagination, while simultaneously getting people accustomed to ‘self-service’. Indeed, many banks are cutting branches, and many new forms of financial services are found only online, like digital banks Fidor and Atom. Digital banking startup Kreditech claims that bank branches won’t exist 10 years hence, “and neither will cost-intensive, manual banking processes”. “We believe algorithms and automated processes are the way to customer-friendly banking,” the startup declares confidently.
Such digital banking is but one strand in the digital trajectory. Digitisation is starting to be applied to more specialist areas of finance, too, such as wealth management. Wealthfront, for example, now offers automated investment advice for wealthy individuals. In their investment white paper they state that sophisticated algorithms can “do a better job of evaluating risk than the average traditional advisor”.
Digital systems like Wealthfront are often promoted as cutting out the middleman – assumed to be human, slow, incompetent and corrupt – and therefore as cutting costs in both money and time. Some startups use this to build a narrative of the ‘democratisation of finance’. Quantopian, a system for building your own trading algorithms, comes with the tagline: “Levelling Wall Street’s playing field”. Robinhood draws on the name of the folk hero to pitch their low-fee mobile stock-trading system.
It seems uncontroversial that these systems may individually lower costs to users in a short-term sense. Nevertheless, while startup culture is fixated upon using digital technology to narrowly improve short-term efficiency in many different business settings, it is woefully inept at analysing what problems this process may accumulate in the long term. Payments startups, for example, see themselves as incrementally working towards a ‘cashless society’, a futurist buzzword laden with positive connotations of hypermodern efficiency. It describes the downfall of something ‘old’ and archaic – cash – but doesn’t actually describe what rises up in its place. If you like, ‘cashless society’ could be reframed as ‘a society in which every transaction you make will have to be approved by a private intermediary who can watch your actions and exclude you.’ It doesn’t roll off the tongue very well, and alarms the critical impulses, but nevertheless, that’s what cashless society would bring.
Part of the reason for the pervasive acceptance of these developments is the deeper ideological narrative underpinning them, one which is found within the tech industry more generally. It is the idea, firstly, that the automation of everything is inevitable; and that, secondly, this is ‘progress’: a step up from the inefficient, dirty services we have now. In this context, questioning the broader problems that might emerge from narrowly useful automation processes is ridiculed as Luddite, anti-progress or futile.
Of course, ‘progress’ is a contested term. If you’re cynical, you may see it as shorthand for ‘the situation an organised set of commercial interests view as desirable in the short-term’. It doesn’t necessarily mean ‘the thing that would be good for the broader public in the long term’.
Indeed, it is apparent that many people don’t respond to ‘progress’ in the way they’re supposed to. We still find people insisting on queueing to use the human cashiers at big supermarkets like Tesco, rather than diligently queueing up for the automated checkout. Likewise, we still find people stubbornly visiting the bank branches, making manual payment requests; even sending cheques.”
* The dark side of digital finance:
“One key to developing a critical consciousness about technology is to realise that for each new innovation a new trade-off is simultaneously created. Think about the wonderful world of digital banking. A low-level bank branch manager might be subservient to the centralised system they work for, but can also deviate subtly from its rules; and can experience empathy that might override strict economic ‘rationality’. Imagine you replace such an individual with an online query form. Its dropdown menu is the digital equivalent of George Orwell’s Newspeak, forcing your nuanced, specific requests into blunt, standardised and limited options. If your problem is D, a system that only offers you solutions to A, B, or C is fundamentally callous. A carefully constructed user complaints system can build an illusion of accountability, while being coded firmly to bias the interests of the company, not the user.
Indeed, if you ever watch people around automated self-service systems, they often adopt a stance of submissive rule-abiding. The system might appear to be ‘helpful’, and yet it clearly only allows behaviour that agrees to its own terms. If you fail to interact exactly correctly, you will not make it through the digital gatekeeper, which – unlike the human gatekeeper – has no ability or desire to empathise or make a plan. It just says ‘ERROR’.
This turns out to be the perfect accountability and cost cushion for senior corporate management. The responsibility and energy required for dealing with problems gets outsourced to the users themselves. And lost revenue from unhappy customers is more than compensated by cost savings from automation. This is the world of algorithmic regulation, the subtle unaccountable violence of systems that feel no solidarity with the people who have to use it, the foundation for the perfect scaled bureaucracy.
So, in some future world of purely digital banking we find the seeds of a worrying lack of accountability and an enormous amount of user alienation. The loan you applied for online gets rejected, but nobody is there to explain what hidden calculations were done to reach that decision. To the bank management, you are nothing more than an abstract entity represented by machine-readable binary code.
Of course, the banks don’t want you to feel like that. In the absence of employees, they will have to use your data to create the illusion of some type of personally tailored service. Your historical interactions with the system will be sold back to you as a ghostly caricature of yourself, fed through the user-experience filters. And it is here that we find the emergence of new forms of financial artificial intelligence.
So how should one respond? One approach is to ride with the technology, rather than to resist it. In intellectual leftwing circles the accelerationist sect advocates an embrace of automation, standing against sentimental calls for more human, local systems. It’s an abstract position, founded on beliefs that automation will create conditions ideal for the downfall of capitalism. At some point it intersects with the cult of the Singularity, popular among evangelical tech entrepreneurs and transhumanists.
The ideological ambiguity is perhaps most acute in the emergent field of blockchain technology. Such systems potentially offer a way for strangers to freely interact with each other without central human intermediaries getting involved in the process. They may use blockchain systems to issue shares, enter into insurance contracts and form digital co-operatives, but the systems are underpinned by an extreme version of automation, one that is essentially autonomous. Indeed, the deep-level mission of projects such as Ethereum, a decentralised platform for ‘trustless’ transactions, is the replacement of human systems of institutional trust – like the legal and political systems that normally underpin all contracts and markets – with automated ones apparently detached from the human ambitions of those who historically have run such systems (‘the politicians’, ‘the regulators’, ‘the bankers’).
Libertarians long for an automated ‘Techno-Leviathan’ to replace the human sovereigns we have now, but it is a big question as to whether such automated systems truly provide a more ‘democratic’ infrastructure for interaction.
More down-to-earth are those who want to allow more creative interaction with the existing digital infrastructure. Take the Open Bank Project, for example, which wants to facilitate third-party customisation of digitised banking processes by opening up bank APIs, in the same way that independent developers might build third-party Twitter apps that draw data from Twitter’s API.
And, finally, we have those who authentically seek to harness digital technology to bypass and challenge the standard economic rationality of large scale, short-term profit-seeking financial beasts, taking advantage of the lower startup costs of a digital setting to promote peer-to-peer finance, alternative currencies, crowdfunding platforms and non-monetary sharing platforms.
So, the scene is set. One thing is for sure: if the future of banking is going to be digital, we want it to be populated with those who value the deeper tenets of open source philosophy. Otherwise we could be left with increasingly alienating, exclusive and unaccountable financial surveillance states, presiding over increasingly passive and patronised users.”