Apple's smart explosion: GPT-4o is added, all family buckets are equipped with generative AI, and Siri is reborn

Not just Siri or the iPhone, but Apple as a whole, has taken a big step forward.

At the early morning of June 11th, Beijing time, Apple's Worldwide Developers Conference WWDC was officially held at Apple Park in Cupertino. At this conference, Apple finally brought a generative model covering all its products as we wished. AI Technology, and at the same time some unexpected news.

"Apple's goal has always been to build people-centric, easy-to-use and convenient personal devices to improve people's lives. For many years, we have been applying artificial intelligence and machine learning technologies to achieve our goals," said Apple CEO Tim Cook. AI Breakthroughs give us the opportunity to push the experience to new heights.”

Now we finally know how Apple thinks about generative AI. First, the philosophy: it must be powerful, intuitive, fully integrated, personalized, and private.

Then the method: Based on the powerful M series chips of Apple devices, we adopt the strategy of self-developed local large models plus cloud computing. The local model is unconventional. Problems that exceed local processing capabilities can be solved with the help of large cloud models (Private Cloud Compute), or you can look for OpenAI's GPT-4o.

Thirteen years ago, Apple's voice assistant Siri was born, exploring the next interactive revolution for smartphones. In the era of generative AI, Siri finally has the opportunity to fulfill people's high hopes. It has become smarter and has rich knowledge. It will give you step-by-step prompts and help you solve problems like the most advanced large model tools currently available.

Also, now like ChatGPT, you can also type to Siri.

Apple said that Siri's new form will be a game changer. A large number of new AI capabilities will be launched soon, and other capabilities such as screen reading and operations within and between apps are expected to be in place next year.

The new Siri is only a small part of Apple's AI capabilities. In this year's WWDC keynote, which lasted for an hour and a half, Apple listed AI capabilities in a separate chapter for the first time, specifically introducing generative AI from images to text, covering mobile phones, iPads and Macs. They are all based on Apple Intelligence.

Apple intelligence: Complete AI system

Apple intelligence is Apple’s new personalized intelligent system that fully integrates the capabilities of generative AI.

Apple Intelligence combines generative AI models with the user's personal profile to provide practical intelligent services. It covers iPhone, iPad and Mac, and is deeply integrated into iOS 18, iPadOS 18 and macOS Sequoia. It uses the power of Apple chips to understand and create language and images, can perform operations across applications, and use personal information to simplify and speed up daily tasks.

These applications can run on the device, and the parts that exceed the device's capabilities can also run in the cloud. With Private Cloud Compute, Apple has set a new privacy standard in the field of AI, with the ability to flexibly adjust computing power between on-device processing and large server-based models running on dedicated Apple chips.

Cook said that Apple Intelligence is a new chapter in Apple's innovation and will change the way users use products. He emphasized that Apple's unique approach combines generative artificial intelligence and users' personal information to provide truly useful intelligent services. In addition, Apple Intelligence can be used in a completely private andSafetyApple AI helps users access information in a way that helps them accomplish what’s most important to them.

Siri is reborn

Apple Intelligence brings deeper system integration to Siri. Siri now has richer language understanding capabilities, making it more natural, contextual, and personalized, simplifying and speeding up everyday tasks. Siri can understand the user's hesitation when speaking and maintain context between different requests. Users can also communicate with Siri by typing and switch between text and voice as needed. In addition, Siri has a new design, and when Siri is active, the edge of the screen will be surrounded by an elegant glowing effect.

Now, users can type text to Siri, or switch between text and voice to communicate with Siri in any way that suits them.

Siri now provides users with comprehensive device support, answering thousands of questions about iPhone, iPad and Mac operations no matter where they are. For example, users can learn how to schedule messages in Mail, how to switch from light mode to dark mode and more.

And, with screen awareness, Siri makes it easy for you to perform actions related to the information on the screen, such as adding an address received in a chat message to a friend's address book.

With Apple Intelligence, Siri can perform hundreds of new actions in Apple and third-party apps. For example, users can say, "Find that article about cicadas in my reading list," or "Send Malia pictures of Saturday's barbecue," and Siri will automatically handle these requests.

Siri can now perform hundreds of new actions within and across apps, including finding book recommendations sent by friends in Messages and Mail

Siri can provide personalized intelligent services based on the user's device information. For example, users can say: "Play the podcast recommended by Jamie", and Siri will locate and play the podcast without the user having to remember whether it was mentioned in a text message or email. Users can also ask: "When does Mom's flight arrive?" Siri will find the flight details and cross-reference with real-time flight tracking data to provide the arrival time.

Siri provides intelligent services tailored to you based on information you have on your device and on your device, such as finding details about an upcoming flight or tracking a dinner reservation.

In fact, Apple had already revealed this update of Siri in a paper in April, but it did not attract enough attention at the time. For details, please refer to the report of Synced: "Let the big model understand the mobile phone screen, Apple's multimodal Ferret-UI uses natural language to control the mobile phone".

In addition, Apple has also open-sourced some related research, see: https://github.com/apple/ml-ferret?tab=readme-ov-file

Integrate ChatGPT across Apple platforms

As expected, one of the highlights of Apple's event today was its collaboration with OpenAI.

Apple announced that Apple is integrating ChatGPT into experiences within iOS 18, iPadOS 18, and macOS Sequoia, allowing users to access ChatGPT, including image and document understanding capabilities, without having to jump between tools.

In addition, Siri can also take advantage of ChatGPT's professional answers at any time. However, Siri will ask the user before sending any questions, any documents or photos to ChatGPT, and then Siri will give the answer directly.

When the user grants permission, Siri can use ChatGPT's answers

In addition, Apple's system-wide writing tools can also use ChatGPT to help users generate content. Through Compose, users can also access the ChatGPT image tool to generate images in a variety of styles.

Writing Tools Visit ChatGPT to assist with writing

As for the launch time, Apple said ChatGPT will be available on iOS 18, iPadOS 18, and macOS Sequoia later this year, powered by GPT-4o. Users can access it for free without creating an account, and ChatGPT subscribers can connect their accounts and access paid features directly from these experiences.

Finally, Apple Intelligence is completely free for users and will be available in English beta as part of iOS 18, iPadOS 18, and macOS Sequoia this fall. Broader features, software platforms, and other languages will be available next year. Apple Intelligence will be available on iPhone 15 Pro, iPhone 15 Pro Max, iPad and Mac with M1 and later.

This means that if you want to use these large model capabilities, you have to spend money to buy the latest Apple devices.

New language understanding and creative abilities

Apple Intelligence unlocks new ways for users to write better and communicate more effectively.

New system-level writing tools are built into iOS 18, iPadOS 18, and macOS Sequoia, allowing users to rewrite, proofread, and summarize text in almost any writing environment, including emails, memos, pages, and third-party apps.

With the Rewrite feature, users can choose from multiple versions to adjust the style to suit different readers and occasions. Whether it's adding persuasiveness to a job application letter or injecting humor and creativity into a party invitation, Rewrite can help users find the right expression.

The proofreading function deeply checks grammar, vocabulary and sentence structure, and provides suggested editing opinions and explanations, allowing users to easily review or quickly adopt them. For example, when a user is composing an email, the writing tool menu will pop up, providing options for proofreading and rewriting, and users can choose the corresponding function according to their needs.

The summary function allows users to select text and generate concise and clear paragraphs, bullet points, tables or lists with one click, making the information clear at a glance. For example, when users start the memo app and record content related to comprehensive health, they can use the "summary" function to extract the key points.

Managing emails is a headache, and the new "Priority Mail" feature puts the most urgent emails - such as dinner invitations or boarding passes for the day - at the top of the inbox, allowing users to see a summary of each email at a glance without opening the email.

When faced with a long email thread, users can get key information by simply tapping the screen.

The smart reply function provides quick reply options and can accurately identify the issues in the email to ensure that each one is properly responded to, making email managementXiaobai NavigationThe management is more handy.

This deep understanding of language also extends to notifications, with the most important notifications promoted to the top of the notification list, and snippets that help users quickly scan long or stacked notifications on the lock screen to surface key details.

Reduce Disturbance is a new focus mode that helps users better focus on the task at hand when a group chat is particularly active by showing only notifications that might require immediate attention, like an urgent text to pick up your child early from daycare.

In addition, a new feature has been added to the Notes and Phone apps, allowing users to record calls, transcribe the conversation in real time, and automatically generate summaries.

During a call, if the user chooses to record, all call participants will receive a prompt. As soon as the call ends, Apple Intelligence will immediately generate a summary to help users quickly review and grasp the key information in the conversation.

Image Playground

Apple Intelligence provides exciting image generation capabilities that can help users communicate and express themselves in new ways, mainly reflected in the new application function Image Playground. With Image Playground, users can create interesting images in seconds and choose from three styles: animation, illustration or sketch.

Image Playground is easy to use and built directly into apps including Messages, and is also available in dedicated apps, making it great for trying out different concepts and styles. All images are created on the device, and users can try out as many images as they want.

With Image Playground, users can:

• Choose from a range of concepts in categories such as Theme, Clothing, Accessories and Location;

• Enter a description to define the image;

• Select someone from your personal photo library to include in the image;

• And choose the style you like best.

With the Image Playground feature in Messages, users can quickly create fun images for friends and even see personalized suggestions related to their conversations. For example, when a user sends a message to a group about hiking, they will see suggested concepts related to friends, destinations, and activities, making image creation faster and more relevant.

In Notes, users can make their notes more visually appealing by accessing Image Playground through the new Image Wand in the Apple Pencil tool palette. Rough sketches can be turned into pleasing images, and users can even select blank space to create an image using the background of the surrounding area.

Additionally, Image Playgrounds can be used in applications such as Keynote, Freeform, and Pages, as well as third-party applications that use the new Image Playground API.

Genmoji: Taking emoji to a whole new level

Users can create original Genmoji to express themselves, simply enter a description and Genmoji that matches the requirements will appear with other options.

Users can even create Genmoji for friends and family based on photos. Just like emoji, Genmoji can be added inline to messages or shared as stickers.

Users simply enter a description to generate a Genmoji along with other options.

Just like emoji, Genmoji can be added inline to messages

New features in Photos give users more control

With Apple Intelligence, searching for photos and videos will become more convenient. Users can use natural language to search for specific photos, such as "Maya skateboarding in a tie-dye shirt" or "Katie with stickers on her face."

In addition, video search has become more powerful, allowing users to find specific moments in a clip and jump directly to the relevant clip. In addition, the new "Clean Up" tool can identify and remove distracting objects in the background of a photo without changing the subject.

With the "Memories" feature, users can create stories they want to watch by simply entering a description. With language and image understanding, Apple Intelligence will pick the best photos and videos based on the description, create storylines based on the themes identified in the photos, and arrange them into movies with unique narrative arcs. Users will even receive song recommendations from Apple Music to match. As with all Apple Intelligence features, users' photos and videos will remain private on their devices and will not be shared with others.

A new standard for AI privacy

For Apple Intelligence to truly help users, it must understand deep personal context while protecting user privacy. The cornerstone of Apple Intelligence is on-device processing, with many models running entirely on-device. For more complex requests that require more processing power, Private Cloud Compute combines the privacy andSafetyExtends the capabilities to the cloud to unlock more intelligent features.

With Private Cloud Compute, Apple Intelligence is able to flexibly scale its computing power and utilize larger server-based models to handle more complex requests. These models run on servers powered by Apple chips, providing Apple with a foundation to ensure that data is never retained or exposed.

Independent experts can examine the code running on Apple chip servers to verify privacy protections. Private Cloud Compute ensures that iPhones, iPads, and Macs do not communicate with servers through encryption unless their software has been publicly documented for inspection. Apple Intelligence with Private Cloud Compute sets a new standard for privacy protection in the field of AI, providing users with trustworthy intelligent services.

Andrej Karpathy: Apple's smart phone is very exciting

Apple's Apple Intelligence has attracted the attention of technology professionals around the world. Andrej Karpathy, a founding member of OpenAI, wrote a post summarizing that he really likes the "Apple Intelligence" released by Apple. He personally observed the following themes:

1. Multimodal input/output. Apple has enabled text/audio/image/video reading and writing. It can be said that these are native human APIs.

2. Agentic. Apple allows all parts of the operating system and applications to interoperate through "function calls"; the kernel process LLM can arrange and coordinate the work between them based on user queries.

3. Frictionless. Apple has fully integrated these features in a highly fluid, fast, always on and contextual way. No need to copy and paste information everywhere, prompt engineering, etc. The user interface has also been adjusted accordingly.

4. Proactive. Instead of performing tasks in response to prompts, Apple anticipates prompts, makes suggestions, and proactively performs tasks.

5. Tiered empowerment. Move as much intelligence onto the device as possible (Apple silicon is very helpful and appropriate), but allow optional offloading of work to the cloud.

6. Modularity. Allows OS access and support for the entire growing LLM ecosystem (e.g. ChatGPT announcement).

7. privacy.

Karpathy said we are rapidly moving into a world where you can turn on your phone, say anything, and it responds to you, it understands you, and it just works, which is very exciting.

The new macOS system enables continuous interoperability between Mac and iPhone

This time, Apple's macOS system also ushered in a major version update, including the system name, a series of new features, etc.

macOS 15 has been renamed macOS Sequoia, and will be released as a public beta next month, with a full version for general users this fall. The most notable features introduced include iPhone mirroring, Notifications, and Safari upgrades.

We focus on the newly added iPhone mirroring feature, which allows users to fully access and use iPhone directly on Mac. Users can launch and browse any iPhone application they want to perform operations on Mac devices, and interact with the phone seamlessly through keyboard, trackpad and mouse.

Slide the iPhone screen.

Open the iPhone app.

With the iPhone notification feature on Mac, users can receive iPhone notifications on Mac and click on the notification to enter the corresponding application.

When the user is working on the Mac, the iPhone screen will be locked in standby mode and others cannot access or see what you are doing.

Easily transfer files between Mac and iPhone by dragging and dropping on Mac.

Although iOS apps can already run on Mac, it is obviously more intuitive to operate the mobile phone interface directly. In this regard, Mac can be said to have kept up with the pace of Android and Hongmeng.

macOS Sequoia also adds a new window arrangement tool similar to Windows, which can automatically adjust the size of the application window to tile and fill the screen. When the user drags the window to the edge of the screen, the system will automatically suggest where to put it on the desktop, so that the desktop is tidy.

Users can choose to tile windows side by side or in a corner to see more apps, while new keyboard and menu shortcuts can help arrange tiles faster.

In addition, the Mac's built-in browser Safari makes it easier for users to discover information, such as routes, summaries, or quick links, through the "Highlights" feature. Machine learning technology is used here to automatically detect the information the user is browsing and highlight it.

Highlighted when planning a route.

It can be seen that the Mac experience under the new macOS system is easier, more convenient and more efficient.

Going against ancestral teachings and installing a calculator on the iPad

For iPad users, the biggest improvement is that there is finally a native calculator app. Jobs once said that putting a calculator on the iPad was "counterintuitive", so for more than a decade, this world's most popular tablet device has not had a calculator app.

Now that generative AI has arrived, Apple has immediately "violated its ancestral teachings."

Apple has introduced a new Math Notes calculator that enables users to type or handwrite math expressions and immediately see the results solved in their own handwriting. Users can also assign values to variables when learning new concepts, calculating budgets, and more. New drawing features allow users to write or type equations and insert charts with just a tap, and even add multiple equations on the same chart to see the relationship between them.

Designed for the unique capabilities of iPad, the Calculator app offers a whole new way to solve expressions using Apple Pencil.

Of course, the premise of all this is that you have to have an Apple Pencil.

This basic scientific calculator on iPad makes it easy to see complete expressions before completing them. A history feature helps users keep track of previous calculations, while unit conversion allows users to quickly convert units of length, weight, currency, and more.

With Math Notes, the calculator allows users to type or write mathematical expressions and immediately see their solutions, as well as assign values to variables for use in expressions.

One More Thing

In addition to the major upgrades to macOS and iPadOS, Apple has also updated the systems on other devices. The mixed reality headset Vision Pro has a new system - visionOS 2, which adds many new features, such as using advanced machine learning to derive left and right eye views from 2D images and create spatial photos with natural depth.

On June 28, Vision Pro will be first released in China, Japan and Singapore. In terms of the price of the domestic version, the 256GB version starts at 29,999 yuan, the 512GB version starts at 31,499 yuan, and the 1TB version is priced at 32,999 yuan. At this price, are you going to buy it?

Apple's series of releases not only keep up with the pace, but also play to Apple's own advantages in hardware and software integration. After all, in the Android camp, it is difficult to see mobile phones and servers using the same chip architecture in the short term. On the other hand, in-depth cooperation with OpenAI, which has the most advanced technology, is also considered to be "open".

So is the prospect of Apple's AI landing bright? Not necessarily, as the stock price still fell today.

After the WWDC Keynote ended, Apple's market value was once again surpassed by Nvidia.

In addition, Musk also said that since Apple is integrating OpenAI at the system level, it is unacceptable.Safety, which is disabled in my company.

NoBoth investors and competitors have some concerns.

Whether "Apple intelligence" can enable Apple to overtake others in the field of generative AI may still need the test of time.

The article comes from the Internet:Apple's smart explosion: GPT-4o is added, all family buckets are equipped with generative AI, and Siri is reborn

Related recommendations: Dialogue with Puffer: How can LRT attract top capitals such as Binance Labs and Franklin to invest in it when it is in the process of internal circulation?

Committed to becoming the base layer of Ethereum. Interviewer: flowie, kit, ChainCatcher Guest: Amir, Founding Contributor of Puffer Finance Editor: Marco, ChainCatcher In the current competitive incentive of re-staking narrative, Puffer Finance is not the fastest player...