Optimize user experience and more with intelligent system software solutions – Samsung Global Newsroom
Following Episode 3
In this relay series, Samsung Newsroom features technical experts from Samsung R&D centers around the world to learn more about their work and how it directly improves the lives of consumers.
The fourth expert in the series is Evgeny Pavlov, head of the Advanced System Software Lab at Samsung R&D Institute Russia (SRR). After 9 years of work dedicated to advanced program analysis techniques at SRR, Pavlov was appointed head of his laboratory in 2020.
The systems Pavlov works on, System Software (SW), is software that has been designed to provide a foundation for other software, such as the operating system (OS) you use in your smartphone, frameworks for AI-powered applications, developer tools and more. System SW is responsible for the communication between the applied software and the hardware. Read on to learn more about the crucial research Pavlov and his team are undertaking at SRR.
Q: The results of research into AI and machine learning are of utmost importance in the design and optimization of all kinds of technologies. What role does research into system software play in further enabling these technologies?
System SW search now plays a very important role in machine learning, although this is not always visible to the end user. First of all, machine learning frameworks do not always perform best on general purpose hardware and processors, so they need to be optimized to take into account various hardware features and use unit extensions. additional central (CPU).
Additionally, the latest trends in the artificial intelligence (AI) industry include the integration of specialized processing units for neural network acceleration. Many companies have recently developed specialized accelerators for neural networks called neural processing units (NPUs). For the optimal processing of a machine learning model, it is necessary to transform the neural network model into a set of instructions for this accelerator.
These neural network model conversions are typically automated using a neural network compiler, as a deep understanding of the NPU architecture is required for the development of these compilers. This means that we, the developers of system software, are involved in their development because we have a deep understanding of how computer hardware works.
In other words, with this change in industry requirements, the focus of System SW engineers is shifting from optimizing general purpose programs to optimizing programs based on AI and machine learning.
Q: Can you please briefly introduce the Samsung R&D Russia Institute (SRR) and the type of work that takes place there?
These days, at SRR, we are focused on developing our expertise and capabilities in three main areas of R&D: Sensor Solution, AI Imaging and System SW. SRR has end-to-end experience in sensor R&D, which includes hardware and algorithm development as well as commercialization specifically for biometric and life care solutions. SRR has been deeply involved in the development of iris, face, and fingerprint biometrics as well as body composition estimation for smartwatches. SRR has also contributed to enhancing the well-known functionality of Super Slow Motion and Night Mode on smartphone cameras by constantly developing the synergy between optics and AI in the field of AI imaging.
I believe System SW is one of the most promising areas of research in SRR at the moment. Based on our in-depth understanding of various hardware and operating systems (OS), as well as a strong engineering workforce, we are doing our best to be a System SW core technology provider for the whole company.
Q: Following your accomplishments in SRR’s Advanced System SW Lab, what are you working on at the moment?
We are conducting extensive research on potential new directions for our System SW team to understand the latest trends in System SW that may well replace traditional System SW techniques in the near future.
Our lab is also currently working on a project to enable 5G scalable vRAN infrastructure to support multiple types of networks, as well as other projects related to compiler technologies for Android and Tizen OS, advanced OS development and the development of software development kits (SDKs). for the AI on the device.
In addition to leading the Advanced System SW lab, I am also currently leading an SRR project for the On-Device AI platform called ONE, or On-Device Neural Engine. This project is being developed in conjunction with Samsung Research’s On-Device Lab, and a major aspect of this project is maintained by Samsung as an open source project located at github.com.
Q: On-device AI and advanced system software technologies are essential to provide users with robust and innovative mobile experiences. Could you explain a little more why this is, and the direction of research that you and the Advanced System SW Lab have taken?
System SW plays a key role in the operation of the application and the user experience. System SW is the bottom layer that sits between a device’s hardware and user applications, which means it forms the basis for all other software. Users may not see System SW in action, as their interactions with their mobile apps are relegated to a simple dialog with the interface, but under the hood of their favorite apps are many layers of program logic – for example, manage the recognition of a tap for the screen in the system kernel, then by drawing a corresponding window in the graphics library. If there is a delay in any of these recognition levels, overall system performance is affected and a user’s experience may also be affected. Therefore, System SW includes special requirements for memory consumption and latency.
The possibility of integrating specialized hardware accelerators in mobile devices has already strongly influenced the development of AI-based applications. This integration improves image quality, locking biometric devices, predictive keyboards, and more. – technologies to which users are now so accustomed that it would be difficult to imagine a mobile device without them. Further development of accelerators is expected to make our mobile devices even smarter, easier to use, and will open up new possibilities for AI applications that previously might only have been imagined in sci-fi movies. .
The SW system can also be enhanced by using these AI-based technologies for personalizing a mobile device for a specific user, for example by providing adaptive settings based on location, behavior and patterns of use of user’s device. Our team is actively involved in such research on improving the SW system through the use of AI technologies on the device.
Q: What do you think are the main user benefits of incorporating on-device AI technologies into mobile devices?
AI on the device is a relatively new technology and is closely related to the growing popularity of AI-based applications. Initially, these applications were run using a high-performance cloud server where all the complex calculations were done, but the growth in performance of mobile processors and the integration of specialized hardware accelerators mean that AI applications can now be developed to run directly on a mobile device. , not a server.
Running neural networks on the device for AI applications has a number of benefits for users. First of all, the response time for users taking advantage of their application is reduced, since it is no longer necessary to send data to the server and then wait for the result; second, the confidentiality of user data is preserved because all processing takes place on the device; and third, these applications can work even without an internet connection.
Q: How does your idea development process, both internally and with domestic companies and universities, serve to provide users with better experiences?
At SRR, we are proactive by monitoring the latest trends in relevant fields, organizing internal seminars, exchanging experiences, interacting with other teams and developing our proofs of concept. This exchange of experiences takes place mainly at informal events, at lunches or in the kitchen, and often gives very interesting results. We also regularly organize brainstorming sessions to generate new ideas. One of the last brainstorming sessions we did was related to the future development of an open source low-level virtual machine (LLVM) project, in which we generated around 30 different ideas, and after filtering, we chose 3 of the most promising areas that I am confident are ready to expand our skills and will be useful for Samsung’s business later on.
In addition to interactions with other teams within SRR, our research center organizes external seminars and joint workshops in which we share our experiences, discuss current trends and share ideas on existing technological challenges. Here in Russia, we are fortunate to have a very strong set of system programmers thanks to the focus on developing System SW at the university stage.
Q: What do you think are the main trends in your industry right now? How did you integrate them into the research you do at SRR?
I think System SW will become more and more optimized with the adoption of machine learning. This will allow us to focus on more complex tasks and get rid of routine optimization tasks. Smart System SW will allow us to achieve the best performance in information processing.
Moreover, AI on the device will not only make our mobile devices smarter, but also our wearable devices, which will ultimately lead to widespread use of AI on all types of devices. Connecting these smart devices will require high-speed communication methods that harness communication technologies such as 5G and beyond, which have the ability to dynamically balance the load between computing nodes in the network. This direction of research is also currently actively explored in our laboratory.
Interview with Ratnakar Rao, expert in advanced communication systems at Samsung R&D India – Bangalore is found in the following episode.