Thursday 15 April 2021

Is Your App Reddit Proof? Robinhood and WallStreetBets are a Warning to All Involved in Software (Reposted)

Rewritten and published in the March 2021 edition of TeaTime with Testers Magazine. Revised version posted here several weeks after magazine publication.

Did you hear about the battle over GameStop between Reddit-based stock traders and big hedge funds during January? This is about how the vulnerabilities and limitations of a popular stock trading app, Robinhood, were exploited by a group of stock gamblers on the subreddit r/wallstreetbets to not only gain profits for themselves but hurt Robinhood financially and eventually strike a blow at the heart of Wall Street finance itself. It is a lesson to devs, testers and company owners that in the ages of social media, financial vigilantism and doing things just to troll, the costs of lack of vigilance and underestimating your user base are far more costly than ever before. Robinhood’s experience also shows that the costs associated with bugs and limitations are amplified infinitely when social media and highly visible trolling are taken into account.
 
Over January 2021 the 4 million members of the retail stock trading subreddit r/wallstreetbets did something utterly incredible that caused major ripples throughout the US financial system. The US bricks and mortar computer games retailer Gamestop had been struggling for several years. The COVID-19 lockdowns then hurt it badly such that it closed 300 stores at a loss of $165 million, its prospects looked bleak and its share price was in the doldrums.
 
Some US hedge funds, notably Melvin Capital and Citron Research decided to take advantage of Gamestop's continuing decline. They did this by short selling its stock - borrowing stock from other financial services institutions to sell and push down the share price, to repurchase later and pocket the difference. The short selling was enormous, with at one point more than 140% of the total shares issued being shorted. Instigated by the update posts of a member of r/wallstreetbets, Keith Gill aka Reddit user u/DeepFuckingValue, members of the subreddit and associated Discord decided that this situation could be reversed by buying cheaper Gamestop (known as $GME) shares or put options (options to buy the stock at a specific price at a later date) to force up the price. Since shares borrowed for short selling have to be bought back and returned by a certain date, the buying by r/wallstreetbets users put pressure on the funds shorting the stock to buy it back (thus forcing up the price further) at a loss. This is a common technique known as a short squeeze. Similar short squeezes were done against the similarly short sold cinema chain AMC and phone companies Nokia and BlackBerry.
 
WallStreetBets users had various motivations. The subreddit, calling itself "Like 4chan found a bloomberg terminal." has a great irreverence of usual risk based trading and investing approaches favoured by Wall Street and an antipathy and meme-based mockery of Wall Street institutions. It also has a culture of heavily leveraged options trading, taking large gambles to make profits (known as "tendies") quickly (and in this case there were great profits to be had), with heavy wins and losses frequently shown off in posts. Members often see themselves as the vanguard of the common trader against the powerful elites of Wall Street, a democratizing force in finance. However other commentators like Bloomberg's Matt Levine put the motivation down to boredom and trolling - just "utter nihilism, a story perhaps best told with rocket emojis".

 

The Robinhood Mobile App and Brokerage


To explain how it provides a warning for testers and tech teams it is necessary to provide some history. The most popular brokerage and trading platform for retail and small traders was the mobile app startup Robinhood. Charging no brokerage fees and allowing small purchases of even fractional shares, it was an extremely easy way for young people with time on their hands and only small amounts in their pockets to get started in the world of share investment and trading. For this reason it was the preferred app for the US-based of the WallStreetBets crowd and heavily used for the short squeeze.

WallStreetBets uses Robinhood to Attack the Hedge Funds 


With the sheer numbers of retail traders piling in (especially after a supportive tweet by Elon Musk) the share price of $GME rocketed from $19.95 per share to $347 per share in just over two weeks. Melvin Capital was forced to close their short position at a 30% loss to their entire portfolio, requiring a $2.7 billion investment by other companies to keep Melvin Capital afloat. Overall an estimated $6 billion was lost by investment firms and hedge funds who were shorting Gamestop and the S&P 500 fell by about 5% over the course of about three days as nerves spread through the market.

Meanwhile Keith Hill's investment of $52 000 in options was worth $42 million by the share price's peak. Robinhood, Brought to its Knees, stops all trading in GameStop The sheer numbers of Reddit traders taking part in the short squeeze also brought Robinhood to its knees. Clearing houses used by the trading platforms started asking for higher amounts of collateral for the trades than the platform could afford. This is important as trades are not instantaneous (usually taking a few days) and usually backed by collateral. Robinhood had to raise $1 billion from its backers and debt facilities to maintain collateral for its trades. Also Robinhood makes 40% of its revenue from a data selling arrangement with the hedge fund Citadel LLC, which part-owns the attacked hedge fund Melvin Capital, and the short squeeze was a conflict of interest that was starting to unravel that agreement.

On January 29th the Robinhood app along with other online trading platforms such as WeBull and IMC Markets took the unprecedented decision to ban or limit trading of $GME and other heavily shorted stocks. Retail investors subsequently turned viciously on Robinhood. Over a hundred thousand poor reviews were given against the Robinhood app on Google Play Store lowering its overall star rating to 1 star, requiring Google to remove them. Criticism of the attacks on r/WallStreetBets and Robinhood's decision came from politicians, media and entrepreneurs across the political divide - Alexandria Ocasio-Cortez tweeting "Gotta admit it’s really something to see Wall Streeters with a long history of treating our economy as a casino complain about a message board of posters also treating the market as a casino" along with Donald Trump Jr. tweeting "It took less than a day for big tech, big government and the corporate media to spring into action and begin colluding to protect their hedge fund buddies on Wall Street. This is what a rigged system looks like, folks! "

What does this have to do with software and quality? The answer is a great deal. This is far from the first time that Robinhood and its app have been put to the test and found wanting by members of WallStreetBets and other new retail traders. The risk of large scale attacks and exploitation via social media opens up a new frontier in what business experts and testers have to watch out for.

 

The "Infinite Leverage" Bug


A much worse problem with the Premium Gold service of the Robinhood app was exploited later by around twenty r/WallStreetBets members in November 2019. As described by Business Insider it involved the following exploit -

"Users who pay a premium for Robinhood Gold sell call options with money borrowed in the app (a loan know as a margin or leverage). Robinhood incorrectly adds the value of the options sold to the user's cash pile. This gives the user more capital to trade with, and the more a user borrows, the more the app adds to their buying power. There seems to be no limit to how much a user can exploit the trick." Call options (in the above case "covered" call options) are contracts that allow the buyer to purchase a stock at a set price at a future expiry date. A seller (or "writer") sells for a fee the right to buy the stock (which they must sell if the buyer asks for it), the hope being that the underlying stock will always remain below the agreed purchase price (known as the exercise price) and thus the option will expire unused - the seller pocketing the cash made from selling the option and retaining the stock.

The bug in this case was that the more the user borrowed to sell call options, the more the app added this to their balance and thus the more the app allowed them to borrow. The original discoverer of the bug, u/ControlTheNarrative, used the flaw to write $50000 worth of Apple put options from a $2000 deposit. One user, u/MoonYachts, was able to borrow a margin of at least $1 million for an original sum of $4000! The user u/Cal_Warrior went even further, turning a $3000 deposit into a position of $1.7 million!

They wrote "After seeing people on the almighty wallstreetbets wager a timid 50k or so on average with this new feature available, I thought it was only a clear choice to raise the average for the good of all." Overall about twenty members used the bug to borrow larger sums than were allowed, getting the cue from posts in the subforum. A user u/SocioButt even posted a "Hall of Fame" of users exploiting the bug. It took days for Robinhood to find out and release a patch to fix the bug and communicate with customers and there was no guarantee that it could claim losses from people who used the exploit and lost money. Robinhood also ran the risk of falling foul of regulators such as the SEC and FINRA along with the costs required to take legal action to claim back the funds.

 

Badly Displayed Losses Resulting in the Suicide of Alex Kearns


In June 2020 the student and budding retail trader Alex Kearns tragically committed suicide after seeing a negative cash balance of $730 000 in his Robinhood Margin (i.e. loan) account. According to his family, later that night the company sent an automated email demanding Alex take "immediate action," requesting a payment of more than $170,000 in just a few days.

A note left by Kearns to his family stated the following - “How was a 20 year old with no income able to get assigned almost a million dollars worth of leverage? There was no intention to be assigned this much and take this much risk, and I only thought that I was risking the money that I actually owned. If you check the app, the margin investing option isn't even 'turned on' for me. A painful lesson.”

To compound the tragedy even further it turned out that this was only a temporary cash balance displayed in the app due to an options trade not yet being completed, however this wasn’t evident in the Robinhood app UI to a fairly inexperienced trader such as Kearns. In fact, the day after Kearn’s suicide Robinhood sent an automated email suggesting the trade had been resolved and he didn't owe any money.

Bill Brewster, a relative and analyst at Sullimar Capital, publicly criticised how the app displayed temporary debt exposure, stating “I’d like them to fix the way that they’re showing exposure — I want them to act like a financial platform should act. When you’re dealing with retail money and actively soliciting traders under 30 years old to have errors like this is inexcusable and at the minimum negligence.”

Robinhood responded by offering to make changes to their in app messages and history page to make the mechanics of trading options clearer, along with providing more stringent eligibility requirements and better educational resources for new investors. However William Galvin, chief financial regulator in the state of Massachusetts, found over 600 instances of people in the state who should never have been approved for options trading by Robinhood’s own standards but were. CBS News confirmed how easy it was to get around Robinhood’s eligibility checks by simply “upgrading your experience”.

Alex Kearn’s family have since filed a lawsuit for wrongful death against Robinhood.

 

Implications for Testers, Quality and Risk Management


The badly displayed temporary debt in the UI and poorly written automated messaging created a tragedy for a brand new trader like Alex Kearns. Robinhood app created the situation where easy access to risky options trading resulted in the tragic consequences as well as permanently damaging the company's reputation. That such a thing was allowed to happen and not flagged up by Robinhood’s internal processes is nothing short of disgraceful and a moral failure.

One way that could have improved the interface such as to prevent the above would have been to apply persona based tests - testers creating personas to study the app interface, emails and warning messages from the perspective of new retail traders lacking experience and financial expertise.

The "Infinite Leverage" flaw in particular highlighted the speed in which bugs are made public and exploited in online forums along with the motivations in which anonymous exploiters use the bug to one up each other online. Suddenly issues that may carry one risk if an individual does it are much graver when social media is taken into account and lots jump on the bandwagon. They also carry new reputational and regulatory risks when forum posts go viral and are reported in the press. In effect, brokerages and companies reliant on traders in groups like r/wallstreetbets need to be aware that the spotlight is always on them and mistakes and errors will be found out and the word spread quickly. The costs of failure are thus potentially enormous and testers and developers working on these apps have to always be "on the ball". They also need real understanding of the users coming to their apps, along their levels of experience, and the social media worlds they inhabit and are influenced by.

The lesson gained from r/wallstreetbets and other groups of small retail traders in their Gamestop short squeeze is that they are realising their immense latent power and acting in ways that institutions on Wall Street would never have predicted. This includes using apps and brokerage tools to make incredible purchases together which makes collusion difficult to prove and police. This does not just affect shorting hedge funds but the tools they use - online brokerage apps now need to allow groups of small retail traders to make large moves en masse at individual stocks and always have the collateral to manage it, otherwise be punished by these same users.

For the rest of us, this is a parable about the power of social media to allow groups of ordinary individuals to troll and exploit - whether it be as anger against the elites, for financial gain or simply because they were bored and it is a funny thing to do. It is a lesson in that just because ordinary people take part in an activity or use your service doesn't mean you control them, predict what they will do or think they will act (in your definition of) rationally. We have to think again about what we expect of users and the online communities they dwell in. For those of us making and testing products to be used by the masses, this is a wakeup call to all. *Thanks to the great editing work and support of JeanAnn Harrison, without whom this article would have been a poor shadow of itself.

Sunday 30 August 2020

QA Research and Tackling The Causes of Defects - A CEO's Perspective

I often find it difficult to articulate what testing, bug detection and quality assurance means at the C-suite level. I know for a fact that I am not the only one and the inability to articulate our benefits to the bottom line is a big problem among testers. We know that it reduces cost of rework and increases customer satisfaction (if done properly), however how do you quantify or articulate that to the people at the top? Also, would the CEO of a large company, say a multinational, really understand what tackling bugs and why they happen does not just to the health of products and services but also to the overall health and strategy of the business? Would the CEO understand why it is a priority deserving of major R&D, and be proud to discuss it with investors?

Then I found an example of a CEO who, for want of a better word, "gets it". I watched a recent interview with Richard White, the CEO and founder of the Australian logistics software company WiseTech Global. The part where he talked about quality enhancements (12:00 onwards) as their major R&D factor is impressive.



What made this stand out to me was that when asked about R&D and having the edge on competitors, he spent time talking about the continual work done with his CTO to find out why defects occur and waste occurs in software development. He was proud of their continual work in "squeezing out" the failure points and cleaning up technical debt (something that I have never heard a CEO even mention in an investor interview, and something that is de-prioritized often in favour of working on new features).

To me, the most powerful thing he said was that he saw their investment on the above and the plummeting defect rates as providing a higher "yield" on each dollar spent, using the example of a 32% yearly increase in product updates done with just a marginal increase in staff.

This is a tech company thus software development is the core of their work, however there have been more than a few tech companies that have neglected quality assurance. What this shows is a CEO and technology team who see QA as a major priority to the bottom line and have taken steps to quantify it, allocate R&D resources to drive defect rates down and then be able to present and proudly talk about it on a show aimed at investors.

If a CEO can articulate the value that working on QA and defect resolution and prevention can talk about it so proudly, precisely and articulately, we testers should too.

Thursday 3 January 2019

(Reprint) Bullying in Tech - It Has to Stop

(A reprint of an article I wrote for Testing Trapeze Magazine, published in its final issue in Dec 2017. Published on this blog as I still regard it among one of my best pieces of writing and its message remains as important as ever. Thanks to Katrina Clokie and Sarah Burgess for their editing).
From Pixabay, CC0 Creative Commons
"I see you're putting on a bit of weight, aren't you Paul!"

"Beg your pardon?" I turned around to face my boss at the time, who was standing behind me.

"You're getting fat aren't you."

I was nervous. How should I react to a senior manager making a statement like that? Do I argue back, remain silent, jokingly agree with him?

"If you say so..." I returned to work.

He said it again a few weeks later. One of many underhand comments about my weight. His power over me at an early stage of my work life, the power to determine if I was employed or not - if I could pay my rent or not - was enough to make me keep my mouth shut.

Some years later, in another team, I was frequently berated by some of the developers. Anger and open ridicule if I made a mistake (especially in front of my two colleagues as test lead), sarcasm in front of colleagues, commenting about and mocking the words I used in conversation. In the workplace I deemed it utterly unprofessional and unacceptable. However it took me several months to approach the project manager about it (who tried valiantly to stop it, however even he wasn't totally successful) and even longer to call it the "B" word.

I am by far not the only person who has had to deal with workplace bullying. The union Unite, based in the UK, reported the following in a ComputerWeekly 2008 article.

"One woman quit after she received e-mails last thing at night with work to be done for the next morning, was given impossible deadlines to meet, and was ridiculed in front of her colleagues."

A 2014 article on the HR website IDGConnect.com describes the story of another victim, "Alex", as follows...

“It was quite insidious... The odd comment here or there. And he’d work his way through the team. Then he started on me and I stood up to him… and it got really ugly. Really ugly - to the point where I went and got a lawyer.”

Apparently, it took "Alex" five years to start to overcome the mental trauma of her mistreatment.

These are not isolated cases. The scale of bullying and unfair treatment in the tech industry destroys health and careers, costing the industry billions in reduced performance, sick leave and staff turnover. We should be doing better.

The Scale of the Problem

Various surveys, and high profile bullying and staff mistreatment scandals over the past 20 years paint a gloomy picture.

The first survey I could find was written in 2002. 3500 UK staff were surveyed by Mercer Human Resources Consulting. 21% of respondents claimed they had been bullied at least once in the past year, with 7% of those suffering chronic bullying. Of those bullied at least once, 24% were middle managers and 17% were senior managers. Those lower down the management chain were more likely to be bullied. In reference to bullying flowing downhill, Mercer’s Patrick Gilbert states, "The high rate of bullying among managers is a particular area of concern. If managers are the victims of bullying, they are more likely to bully the people they manage."

In the previously mentioned Unite survey of 860 UK IT professionals, 65% believed they had suffered workplace bullying. Examples of the types of behaviours reported in the survey were -

"..unachievable deadlines, excessive monitoring and supervision, and constant criticism on minor matters. More than half said they had been bullied by a more senior member of staff."

"The survey found many victims felt unable to report bullying because of fears that it would get worse, or because they thought their complaints would be dismissed as an inability to cope." A wall of silence had developed with victims feeling unable to speak out.

The Kapor Centre for Social Impact's 2017 Tech Leavers Study surveyed  2006 workers in the US who had left a technology-related job in the past three years. It included questions on bullying, public humiliation, and rude and condescending behaviour, with a focus on minority groups.

Taken from the survey -

"LGBTQ employees were most likely to be bullied (20%) and experience public humiliation or  embarrassment (24%), both at significantly higher rates than non-LGBTQ employees (13%, p<.01).

White and Asian males experienced bullying (16%), public humiliation (16%), and rudeness (25%) more frequently than underrepresented men (9%, 11% and 19%).

Bullying and hostility were most often perpetrated by senior-level employees (53%)"

Uber


The culture of bullying and harassment among some Silicon Valley startups, particularly Uber, have been heavily criticised, notably by Margaret Heffernan in the Financial Times . Heffernan described a "frat boy culture of bullying and exclusivity" that limited the prospects and representation of women and minorities. She pointed to the scandals at Uber, which had recently fired 20 of its staff and deposed its CEO Travis Kalanick over 215 claims of discrimination, unfair treatment and sexual harassment. Of these 33 claims related to bullying.

France Telecom


An even more sad and devastating bullying scandal arose at France Telecom (now Orange SA) in 2009, following as many as 35 staff suicides in two years. Whilst France Telecom claimed that the rate of death by suicide was in line with the French average (16 per 100 000 in 2006), many left suicide notes blaming "unbearable work pressure, bullying and 'management by terror'". Alongside this  "scores of other staff, from senior technicians to staff who worked processing bills, were saved as they attempted to kill themselves." Some suicides were attempted in the office during the workday.


"There was this pressure from the top to slim down operations by destabilising workers, people were undermined to the point that they got ill," his sister claimed. "He told me he was regularly sent messages from managers suggesting he find work elsewhere. Once they suggested he open a rural guesthouse. He accepted a far too heavy workload out of fear of losing his senior job."

According to an official report by the Works Inspectorate the blame was laid on a climate of "management harassment" which "psychologically weakened staff and attacked their physical and mental health". France Telecom was restructuring at the time making 22,000 jobs redundant and moving 14,000 jobs to different locations. The mishandling of the moves was seen as a major factor in some of the attempted suicides, a claim denied by the CEO, Didier Lombard.

Lombard, under heavy criticism for his poor handling of the tragedy - particularly his dismissing of the suicides as a "fashion" at the company - was forced to step down from his post in 2010. In 2012 French media published an internal memo from 2006 where he stated in a high level meeting that he would cut staff "...one way or another, through the window or through the door". Following these revelations, in 2016 the Paris Prosecutor recommended that he and other senior figures at France Telecom be put on trial for "moral harassment" (the legal term for bullying in France). According to the BBC, "the Paris prosecutor accuses France Telecom of enacting a policy in 2007 that resulted in unsettling workers and creating a "professional climate that provoked anxiety" at the time of a "delicate restructuring" of the company".

Effects of Workplace Bullying


Both the individual and the technology industry suffer when there is bullying in the workplace.

The Australian Human Rights Commission in its factsheet "Workplace bullying: Violence, Harassment and Bullying" states “If you are bullied at work you might:

  • Be less active or successful

  • Be less confident in your work

  • Feel scared, stressed, anxious or depressed

  • Have your life outside of work affected, e.g. study, relationships

  • Have physical signs of stress like headaches, back aches, sleep problems”

In extreme cases, such as those arising from the toxic culture at France Telecom, the effects  can lead to severe depression, sickness and suicide.

At industry level, according to the Tech Leavers Study 2017, the cost of workplace unfairness (which combines bullying with other damaging practices such as sexual harassment, racial and sexual discrimination in promotion, skill underutilisation, and stereotyping) is  "astounding", both in terms of turnover and reputational cost. Based on an estimate of the turnover cost of a tech employee being $144,000 USD, and the findings that nearly 40% of employees reported leaving their jobs due to unfairness, an estimate to the US tech industry alone is $16 billion per year!

On top of that, according to the study, former employees who were unfairly treated would be 35% less likely to refer someone to that employer, and 25% less likely to recommend buying or using their products.

Compound that with increased sick leave and the risk of litigation and it makes good financial sense to eliminate workplace bullying from our industry.

Stopping Workplace Bullying


Julia Moriarty from The Network - a company that works with clients to stop workplace bullying - states in this article:

“The key to avoiding a negative and disrespectful work environment is to establish and continually support a strong, consistent corporate culture that stops the inclination to bully before the behavior starts.”

"Management has to take the lead. Many organizations don't want to acknowledge that bullying is happening in their workplace - but it is. Keep in mind that a lot of bullying activity may be covert and may not be visible to company leadership until the damage has been done..."

The risk of victims resigning or losing their job is a source of great worry. The article quotes data from the Workplace Bullying Institute that states "56 percent of reported bullies are the victims' boss; 33 percent report that a coworker is the bully, leading many victims to believe that either they'll be fired for reporting the abuse or that their bully will simply ignore their complaints and escalate the abuse."

Moriarty recommends that a “No Tolerance Policy” be instigated and that an executive level position or independent team be allocated to corporate ethics and compliance - reporting to the CEO or board of directors. The position or team must be able to set behaviour guidelines and make objective decisions such as disciplining or dismissal without fear of reprisal or overrule due to office politics or financial concerns. Management also needs to respond quickly to reports of bullying and conduct a transparent investigation with real consequences for those found guilty of bullying.

Managers should receive training on compassionate, effective management skills and techniques. Moriarty states, "All of the training and communication in the world will have no impact if the culture of the organization does not support what is being taught. Managers need to walk the talk and demonstrate respectful behavior themselves. Performance reviews should take into account management style and ethical behavior to ensure managers take the company's standards seriously. The company needs to provide coaching for managers who demonstrate bullying-type management practices and, if they cannot correct their behavior, remove them..."

Peter Skyte from Unite and Cary Cooper from the University of Lancaster offer various tips for managers including, having a strong anti-bullying mission statement and a clear process to be followed if bullying occurs (including escalation beyond the immediate manager if considered part of the problem). They also prescribe a process of swift and decisive action against the bully along with victim support.

What about if you are the target of workplace bullies? The University of Wollongong's Brian Martin offers advice on how bullies work and how to react. His tips are outlined below -

  • Expose the bullying.

  • Validate the target, by demonstrating good performance, loyalty, honesty and other positive traits.

  • Interpret the bullying as unfair, and explain why contrary explanations are wrong.

  • Refuse to be intimidated or bribed, and expose intimidation and bribery.

Interestingly, he recommends being sceptical of reporting to official channels, instead mobilise support in other ways. In his words,

“It's tempting to seek help by using formal processes, for example reporting the matter to the boss's boss or to the board of management, making a complaint using an internal grievance procedure, or making a submission to a review panel. Unfortunately, this seldom helps and can actually make things worse.

People high up in organisations nearly always support the chain of command. A top manager will almost always support subordinates in the face of challenges from lower-level employees."

The Workplace Bullying Institute offers additional advice in what it calls its “3-Step Target Action Plan”. It recommends naming the behaviour as bullying, taking sick leave to heal and decide on a counterattack, and then exposing the bully. Nevertheless, it makes the grim point that in most cases (77.7% of cases in its own statistics) targets of bullying are likely to either lose their jobs or choose to leave the company. It recommends starting the search for a new position.

Epilogue

Incidents of workplace bullying and harassment are causing reputational damage to the tech industry as a whole, and are a blight on the victims. We must come together as a community to stop bullying and the fight against workplace must be supported at the highest levels of the company.

Compliance teams must be given autonomy and power to remove toxic staff and cultures. Victims must be able to have their grievances heard without hurting their own job and career progression.

Stopping workplace bullying is a thing we must do. To protect the wellbeing of workers in our industry, as a moral imperative, and to reduce costs to companies and the industry as a whole.