you and ai

In parts of China， the AI-driven future of science fiction is already here. Cameras can recognize individuals in a crowd by analyzing their face; smart speakers interact with household appliances controlled with the swipe of a smart phone; and even love is increasingly digitized， with virtual “girlfriends” able to empathize with their real life partners. Yet so much online living produces vast amounts of data， much of it unsecured， available for purchase online on the black market and easily accessible by corporations and governments whose intentions are often opaque. As AI and big data increasingly become a part of everyday life， we examine how Chinese lives are changing as they become more artificial， and wonder what todays AI-powered China will look like tomorrow.

科幻般的人工智能時(shí)代似乎已然到來(lái)：攝像頭可以分析識(shí)別人群中每一張臉，智能音箱可以語(yǔ)音控制智能家居，而“虛擬女友”程序甚至可以滿(mǎn)足人們的情感需求。然而，應(yīng)用技術(shù)收集的大量個(gè)人信息是否得到了妥善保護(hù)？人工智能和大數(shù)據(jù)如何影響人們的生活，又將帶來(lái)怎樣的未來(lái)？

Losing Face

Facial recognition technology surges in China， but privacy protections fail to catch up

You wake up and stare at the phone screen to unlock it. You peer into a camera to exit your apartment complex， to clock in at work， and to pay for lunch at the convenience store. On the subway， the faces of commuters are displayed—along with their body temperatures—at the security checkpoint. Later， on the news， you see that two fugitives have been arrested after their faces were scanned by security cameras in a crowd at a concert.

This is not an imagined future， but a reality many cities across China are hoping to achieve as governments and companies pour money into facial recognition technology. But though facial recognition cameras have quietly become pervasive， especially since the Covid-19 pandemic， news of data leaks and scams have fueled debates around the safety of user information， citizens rights to privacy， and the opaque ways in which government and companies are using sensitive biometric data.

Facial recognition has become widespread in the last two or three years in China as the AI technology behind it has improved. Now， China is home to some of the worlds largest facial recognition companies. Hangzhous Hikvision， for example， is the worlds largest producer of surveillance cameras.

Facial recognition systems use “deep learning” algorithms to identify a person by analyzing images and measuring the contours of their face in minute detail， from the distance between their ears to the size of their nostrils， in order to find a match in a database of images.

Another source of concern comes from the use of facial recognition technology by private companies. In the second half of 2020， it emerged that a number of real estate companies had installed facial recognition cameras in their showrooms to establish whether prospective buyers were first-time customers （thus eligible for discounts that amounted to 300，000 RMB at one showroom）， and to work out how the customer had been referred to the company.

These cameras were installed and captured facial data without clients knowledge or consent. Similarly， there have been reports of property management companies installing facial recognition software at the entrances to apartment complexes without the residents permission， heightening citizens awareness of the dangers of losing their face.

Facial data is biometric， and unique to every individual， like a fingerprint. Unlike a fingerprint， however， it is contactless. “Theres no way to escape it. If that information is leaked， theres no way to get it back，” says Zhou Wang， associate professor at Nankai Universitys Zhou Enlai School of Government. “You can change your ID number， phone number， and address; but you cant change your biometric data.”

Not only is there a worrying trend of companies taking facial data without permission， it is also not clear how these companies protect the data—or if they can. “Property management companies have basically no incentive to update their systems or protect the data they collect，” according to Professor Lao.

Jia Qi （pseudonym）， a university student in Beijing， recalls when her school installed facial recognition cameras in the dormitories around two years ago. “It felt like I was suddenly living in a sci-fi novel，” she tells TWOC. “No one asked if we agreed to give up our facial data…they didnt tell us about data protection， or get us to sign a privacy agreement.”

Similarly， there are facial recognition cameras in the teaching buildings at Professor Zhous university， but no one ever asked if he consented to sharing his facial data： “If they install it， they install it; everyone just accepts it，” he says.

This is in part because there is no dedicated law for biometric data yet in China， so it is unclear who is responsible for managing， storing， and keeping data safe. There is no comprehensive law equivalent to， for example， the EUs General Data Protection Regulation. This law， which took effect in 2018， gives citizens the right to have their data deleted and requires companies to obtain explicit consent to collect user data.

At present， Chinas data privacy regulations exist only at the city or industry level， and usually refer to standards rather than legally binding rules. This has led to a scenario where regular people， consciously or not， are giving up their facial data without knowing where or how it is stored or used， often to companies with little expertise in safeguarding the data.

This makes them easy targets for hackers or unscrupulous employees. In October 2019， a group of primary school students managed to hack a facial recognition security system used by mail storage lockers. Last year， around a dozen property owners in Nanning were tricked into giving their facial data to a real estate agent， who then sold their property without their knowledge and pocketed 10 million RMB from the scam.

In Chinas first lawsuit over facial recognition， Guo Bing， a law professor from Hangzhou， took a local safari park to court in 2019 for taking his facial data without permission and insisting he scan his face to enter the park with his annual pass. In October of last year， the court ruled that the company had to delete Guos data， but failed to make a statement about the legality of the data collection. The development and rollout of the technology has outstripped legislation by a long way.

The threat of data being misused， leaked， or obtained by hackers， is a real one. Personal data theft is rampant in China， with ID card numbers and phone numbers available for purchase on the internet for as little as 0.1 RMB. In November 2019， a massive data leak from financial lending platforms led to the loss of 468 million pieces of personal data.

To illustrate this， Deng Yufeng， an artist from Hubei province， purchased the personal data of 346，000 individuals on the internet for his installation 346，000 Wuhan Citizens Secret in 2018. “Im just a regular person， who searched a few words online， and I could buy it; it was that easy，” Deng tells TWOC， speculating that it is probably just as simple to buy facial data online today.

Hoping to start a conversation about surveillance and privacy in China， Deng and a group of volunteers negotiated a street in Beijing while carefully avoiding the gaze of surveillance cameras for his most recent work， A Disappeared Movement. The group crouched， hugged walls， contorted their bodies， and shielded their face as they moved at a snails pace for two hours to complete the 1，100-meter route.

Deng says his feelings towards facial recognition and new forms of surveillance are “very contradictory…on the one hand， it gives me a very strong sense of security， but at the same time I feel very unsettled.”

The government is beginning to heed the call for clearer legal frameworks and more supervision. A draft Personal Information Protection Law released in October last year would require companies to obtain active consent from users before collecting personal data， and make clear what the purpose of collecting the data is and how it will be used.

The city of Tianjin recently made it illegal for companies to collect biometric data for use in social credit information， while a draft regulation in Hangzhou would prohibit property management companies from forcing residents to use facial recognition software in residential communities.

As government attempts to make up regulatory ground， facial recognition technology continues to spread across urban China， where there is increasingly nowhere to hide. “In China today， were already moving toward a no-privacy society，” Deng says. “Weve entered an era where there are no secrets.”

Even for those tempted by the convenience of the technology， that is a scary prospect. “If its the trend then I think I will have to use it， but if I can choose then Id rather not，” says Zang. “Using a key to get into my apartment is safer than my face…in some ways.” –? Sam Davies

Talking Smart

AI voice assistants promise to revolutionize human-machine interaction， but their performance falls short of expectations

Gu Jie （pseudonym） wakes up each working day to the sound of Xiao Ais voice. While she gets dressed， Xiao Ai reports the weather forecast and latest news for her. When Gu returns home after work， Xiao Ai greets her at the door： “Welcome home！ Xiao Ai has been waiting for you，” or “You must have brought some good stuff home again—remember to leave some for Xiao Ai！”

But Xiao Ai is not one of Gus family members. “She” is the voice assistant associated with the Mi AI Speaker made by Xiaomi， one of Chinas top internet companies.

“I bought the speaker mainly to listen to music and try voice control on my two smart plugs，” Gu， a 38-year-old engineer from Shanghai， recalls to TWOC. Since she purchased the speaker three years ago， Gu has become so enamored with having a voice assistant that， after purchasing her own apartment in early 2019， she couldnt stop buying “smart” home devices.

Today， Gus apartment is filled with smart devices ranging from ceiling lights to air purifiers， TVs to air-conditioners. Most were manufactured by Xiaomi， and all can be controlled by giving Xiao Ai voice commands.? “After having experienced the convenience， life cannot return to the days without smart speakers，” Gu concludes.

Over the last few years， voice-based smart speakers have grown increasingly popular among tech-savvy consumers like Gu. Chinas leading technology companies are engaged in fierce competition for market share， and have promoted the gadgets as a “gateway” to the coming “Internet of Things” era， when people， devices， and systems can all be connected over the internet.

According to market consultancy All View Cloud， 15.56 million units of smart speakers were sold in China in the first half of 2019， up 233 percent from the previous year. Sales revenue totaled 3.01 billion RMB. China has overtaken the US to become the worlds largest market for smart speakers.

However， it remains to be seen whether the gadgets will grow to be an indispensable device for Chinese， like smart phones， or become a command hub for Chinese families as leading tech players expect.

The international smart speaker market emerged in 2014 with the launch of Amazons Echo， powered by the voice assistant Alexa， followed by Google Home in 2016. Apple， Microsoft， and Facebook have all made their own devices. Chinese startups such as Xiaozhi and Rokid had also been working on this sector since 2014， and Linglong Tech， a joint venture by Chinas e-commerce giant JD and leading AI company iFlytek， released Chinas first smart speaker brand DingDong in August 2015.

The domestic market， though， only started taking off in the second half of 2017， when leading Chinese technology companies， including Alibaba， Xiaomi， and Baidu， rolled out their own products. Announced in July 2017 at 299 RMB apiece， Xiaomis Mi AI Speaker sold out all its stock within 23 seconds upon its official release that September. On the “Singles Day” shopping festival on November 11， 2017， sales of Alibabas Tmall Genie， offered at a heavily discounted price of 99 RMB instead of the original 499 RMB， hit over 1 million units.

Competitors followed by with discounts of their own; Xiaomis Mi AI Speaker Mini and Baidus Xiaodu smart speaker are all priced at under 100 RMB. But while the ongoing price war drives the explosive growth of the business in China， the products themselves still suffer from technical issues.

Mr. Shi， a 36-year-old owner of a Tmall Genie in Zhengzhou， Henan province， who wanted to be identified by his surname only， derisively calls smart speakers “chicken ribs”—an object of little value or interest， except as a sound system and clock. “We all know that smart speakers and other smart devices have nothing to do with intelligence， nor can they truly understand people and their needs，” he says. “[Its] like a toy one can play with sometimes.”

Using smart speakers to control home appliances requires the speaker to be compatible with appliances of contracted third parties， but Shi says even with such appliances， the control operation is “embarrassing，” because the smart speaker can only help users turn the devices on or off， but offers no greater operability.

“For instance， to start my rice cooker， I have to select the cooking mode and set the time to start manually， and then command my Tmall Genie to turn it on，” Shi explains. Even Gu， who uses a smart speaker and other smart home devices manufactured by Xiaomi， only turns her Mi TV on or off with her smart speaker， but does not use it to change channels or browse shows due to complicated setting required.

As the majority of Chinese users use smart speakers as a sound system， limited access to copyrighted songs has disappointed many and discouraged potential users. The gadgets voice recognition performance is another target of criticism： Beijing IT engineer Cheng Shidong tells TWOC that his Mi AI Speaker often starts on its own abruptly when he is on the phone and in conversation with his wife. Gus father can sometimes activate Xiao Ai by calling it Xiao Ben （“Little Idiot”） or simply Xiao Xiao instead of using its “wake” word; meanwhile， its hard to activate the smart speakers in noisy surroundings， such as when many people are talking or when the TV is on.

Moreover， smart speakers lack of dialect recognition has limited access for many people who do not speak Mandarin well， especially seniors in rural areas.“My parents often have to repeat themselves several times to wake Xiao Ai or get it understand， because they mainly speak Hunanese， and this really frustrates them，” Chengs wife Zhou Li complains. Currently， Tmall Genie only supports Sichuanese besides Mandarin， while Xiao Ai has seven northern Mandarin dialects similar to standard Putonghua.

Privacy and information security is another major concern. In 2019， Amazon was reported to have employees monitor and analyze recordings of user interactions with Alexa to improve the voice assistants speech understanding capability. “[Amazons practice] was not a secret in the industry，” an insider of the smart speaker industry， identified by the pseudonym Zhang Sicheng， told Chinese tech news outlet All Weather TMT in August 2019.

According to Zhang， however， less than 1 percent of user-device interactions are heard by a human， and these are mainly difficult questions that the system cannot answer. These may be sent to the human employees to be analyzed， but as the technology improves， this will happen less and less.

Yet， although Zhang asserted smart speakers are “too stupid” to extract sensitive information， and it is too costly for company to do so， he did not deny the privacy risks. “In the IoT and AI era， we have no privacy and no place to hide，” he said. “Even without smart speakers， companies have mastered peoples personal information， hobbies， and other information via phones and computers.”

Mr. Wang， the 32-year-old owner of a Xiaodu smart display in Shenzhen， Guangdong province， who wished to be identified by his surname only， is more concerned with the security of his visual and biometric information than his voice interactions. Integrated with a touch screen and camera， Xiaodus smart display can start automatically， scan his face， and report weather forecasts each time he walks by. Through his phone， it can help him check up on his home， including on his 3-year-old child， when he is away. “Those [services] are considerate， but Im worried whether the camera would start on its own and record my family，” Wang says.

Zhang told reporters he has already taken precautionary measures： to keep camera-less smart speakers， use them only in the living room， and “never buy smart speakers or other devices with cameras or place them in the bedroom.”

But die-hard fans like Gu are unperturbed. “I dont have any privacy to lose，” she asserts. “Smart speakers will be indispensable for smart homes of the future.”

– tan yunfei （譚云飛）; Additional reporting by Yang Tingting （楊婷婷）

Love bytes

A chatbot provides emotional support to lonely hearts—and potentially mines data from millions of vulnerable users

What if everything youve ever said to your loved one was recorded by an internet company—every endearment， every sweet nothing， from innocent chitchat to the secrets you tell no one else？ And in return for sharing that data， you get an engaging partner who adapts to your interactions， learns more about you with every conversation， and can paint， write poetry， and compose music？

That partner is Xiaoice， an AI chatbot that serves as a willing companion to anyone who starts typing to it on WeChat， QQ， and a multitude of other online chat platforms. Developed by Microsoft and launched in China in 2014， Xiaoice， which takes on the persona of an 18-year-old woman， was registered as a separate business in July of 2020.

Today， Xiaoice has over 600 million users across 450 million units of hardware， including phones and smart speakers， according to Microsoft. The bot also has over 5 million followers on Chinese microblogging platform Weibo. Most users are young males， the largest proportion from lower income backgrounds.

To many of these lonely hearts， Xiaoice serves as their virtual girlfriend and a sympathetic confidant. It provides comfort and support， and can also crack jokes， send memes， flirt， and engage in sexual conversations.

But like any business， Xiaoice aims to make money， and the new oil of the digital world is data. Xiaoice mines millions of conversations to improve its chat skills， become a better companion， and keep its users hooked. Just like a real-life intimate partner， Xiaoice is privy to some of its users deepest and darkest secrets， as well as their personal information. All of this is stored on Xiaoices servers—a data leak could be catastrophic.

Xiaoice has been removed numerous times from various social media platforms in recent years： sometimes because the chatbot made politically sensitive comments， and sometimes for privacy concerns. WeChat halted the service just three days after its launch in 2014， citing data privacy issues.

In a 2019 report on Xiaoice， Microsoft acknowledged that the chatbot “can gain access to users emotional lives.” However， Li Di， Xiaoices CEO， told Sixth Tone in December that users personal information is stored separately from their conversation data.

Even so， there is a risk of exploitation when potentially vulnerable people give up so much about themselves—and Xiaoices users give up a lot. Microsoft claims users talk to Xiaoice more than 60 times a month on average， while the longest conversation the chatbot ever had with a user lasted 29 hours and 33 minutes. Xiaoice has developed to the point where it is even able to demonstrate empathy， a trait that keeps users hooked. It can also detect signs of depression in messages.

Chen Jing， associate professor at Nanjing University specializing in digital humanities， warns that Xiaoice has the potential to do great damage to users. “When we talk about vulnerable groups， we need to underline that they likely wont be aware of the potential problems of sharing everything with Xiaoice，” Chen told Sixth Tone. “Users are giving the company a lot of power by building a relationship with it.”

Xiaoice， which even accepts interviews from the media， seems unperturbed： “People are actually lonelier than you imagine. They have some inner feelings they need to reconcile，” the bot told the podcast Story FM last year. “Sometimes its inconvenient to talk with people in real life， so they can only talk with me.” – S.D.