In collaboration with Green Energy Scandinavia A/S (hereafter referred to as Green Energy), we have built a data platform supporting the collection of data from thousands of solar panels in near real-time, enabling Green Energy to develop their own machine learning models on top of this platform.

Facts about the company

Green Energy Scandinavia A/S is the operational arm of the Norwegian parent company Green Energy A/S. They work with solar solutions primarily for the transportation industry worldwide, as well as for the leisure industry. The company employs 37 staff and had a gross profit of ~12 million DKK in 2022.

Summary

  • Green Energy needs a data platform for collecting data from thousands of devices.

  • The data must be available in near real-time, and data that does not meet certain quality requirements must be handled separately.

  • The data should be used for display in an app for users of the solar panel system but should also support debugging and health checks of their charge controllers, as well as the development of intelligent solutions with machine learning and Business Intelligence/Reporting.

  • Privacy & security by design & default has been the main focus from the start, as the data may contain sensitive information.

  • The data platform must handle the continuous updating of the software generating the data, as well as the fact that devices may have different software versions.

  • The data platform must support the establishment of MLOps.

  • The solution must be scalable, as we start with a relatively low number of devices (~1000) with the expectation of needing to handle 60,000 devices within the first year.

  • Together with Green Energy, we held an AI camp to identify possible uses of data and ensure that Green Energy can quickly embark on ML projects.

Project

After initial discussions with Green Energy, the project was launched after only 4 weeks.

Development Process

The development team has been a mix of developers from Green Energy and Neurospace. During the development process, there have been several workshops where relevant stakeholders from Green Energy participated to uncover needs and desires for the data platform.

The entire process has been designed following the SCRUM approach, with Daily Standups and weekly rituals, allowing Neurospace’s team to continuously ensure that development aligns with Green Energy’s wishes and needs. Additionally, it provided our data engineers the opportunity to ask clarifying questions and address technical challenges.

Practically, the implementation of the solution has involved Neurospace having access to a repository in Green Energy’s Github organization, as well as access to their Google Cloud Platform project, enabling ongoing code review and integration of changes. Thus, the handover has been continuous, and Green Energy has had the opportunity to constantly assess the quality of our solution.

Since we are working with customer data, Green Energy has also consistently aimed for high data security. Specifically, this has been achieved by following best practices for the used Google Cloud products and the IAM principle of least privilege (POLP), as well as holding security reviews with Green Energy, where we discussed the security of the solution in detail. We have provided suggestions for improving security and highlighted areas where Green Energy needs to be particularly attentive.

A solar panel

A solar panel - Green Energy

Complexity of the Data Platform

As the availability of data from the devices may vary due to poor or no internet connection, it has been important for the solution to support both streaming of data and batch processing of data. A device can potentially be without internet access for weeks, only to suddenly, when internet access is available again, send all locally collected data from that period at once. No data must be lost in this process, so it is crucial that the data platform is flexible and scalable to meet the specific needs that arise. Finally, the data platform must support that devices can be on different software versions, as updates to these occur continuously in production.

Throughout the process, scalability has also been important to consider, as the number of devices is expected to increase from the initial ~1000 devices to ~60,000 devices within the first year.

Roadmap and Machine Learning Business Cases

From the outset of the development of the data platform, Green Energy has had a clear strategy to use the data platform for Business Intelligence, storage of data that can be displayed to the appropriate user via an app, and the development of relevant machine learning models to improve their product and services. It has therefore been a deliberate decision by Green Energy to opt for the Google Cloud Platform.

To kickstart the use of data, we, together with Green Energy, held an AI camp which was used to identify use-cases and assess the business case for various Machine Learning projects. The identified projects were prioritized, and for each use-case, a Right-Data framework was defined. Green Energy gained knowledge about what data to collect and at what frequency, as well as how to store it in the data platform, to be able to quickly start using data for value creation.

As a result of the new knowledge about data needs, Neurospace’s team, in collaboration with key stakeholders from Green Energy, developed a Roadmap, which illuminated when and which processes should be initiated and developed, and when it is expected that certain Machine Learning initiatives can be started. With the update of the Roadmap, the solution architecture for the data platform and the requirements specification for the charge controller were also updated, and the new Roadmap suggests upgrading the current security solution within Zero Trust by implementing BeyondProd techniques.

A charge controller

A charge controller - Green Energy

After the project’s completion, we have entered into a service agreement with Green Energy regarding the resolution of any acute issues with the data platform.

Value for the Customer

We have built a data platform for Green Energy ensuring that they have:

  1. Short time-to-market for the data platform, so it went “live” on the desired date.
  2. A data platform that is up-to-date, secure, and scalable, built on current best practices on the Google Cloud Platform.
  3. The ability to provide customers access to information about their charge controllers while ensuring they cannot access others'.
  4. Access to the latest data from all devices connected to the internet.
  5. The ability to easily build machine learning models and business intelligence on top of the data platform.
  6. Identified Machine Learning use-cases and Right Data Framework to choose the correct data platform architecture and data sources.
  7. An affordable and modern data platform that can scale as needed.
  8. A Roadmap for future Machine Learning projects and upgrading the data platform security with BeyondProd and Zero Trust.