Businesses that operate in complex cloud environments, large fleets, or sophisticated security operations all require visibility into their cloud assets in order to keep their teams nimble and their data secure. Cloud Asset Inventory (CAI) helps these teams understand their Google Cloud  and Anthos environments by providing complete visibility, real-time monitoring, and powerful asset analysis capabilities. Today, Cloud Asset Inventory gets four new capabilities that help you understand your environment more clearly and easily than ever before.

New user interface eases asset and insight discovery 

Cloud Asset Inventory console preview is now publicly available for GCP and Anthos customers. This preview provides insights into your cloud footprint, history and details of resource usage with powerful filtering and search capabilities. For example, you can view your global distribution of resources and policies, how your GCE VM footprint has been changing over time, as well as full metadata and change history for all your assets. The CAI console can be filtered at the organization, folder, or project-level, so each user can view the resources they have permissions for down to project level granularity.

Asset discovery and Datadog integration

A new asset list service in CAI provides quick and comprehensive asset discovery, including asset history, without needing to export the data to a storage destination. Datadog, a leading multi-cloud monitoring and security service provider, relies on deep integration with CAI for service and asset discovery. Datadog has been piloting and taking full advantage of the newly released asset list service. Datadog Product Manager, Steve Harrington, commented: 

“Google’s new Cloud Asset Inventory API provides us with an immensely valuable, single source of truth for determining the resources present in a given GCP environment. Along with its rich metadata, this enables us to enhance multiple aspects of our integration with GCP, including streamlined metric collection and ingestion of custom labels. We plan to continue building around Cloud Asset Inventory in the future to improve existing features, and are envisioning ways it could help us provide entirely new insights to our customers.”

Answer “who can access what resources?”

Determining authoritative answers to security-related questions like “Who can read data from my storage bucket that contains PII?” or “Does a terminated employee still have any remaining access to my system?” can be difficult and time consuming. This is why access management and identity certification is one of the top security priorities for enterprises running workloads in the cloud. To help alleviate this challenge, the new Policy Analyzer capability in CAI thoroughly analyzes the relationship between IAM policies and resources. The analysis includes powerful and efficient group expansion, service account impersonation, conditional access analysis, resource expansion, and more. You can even export the results to a BigQuery table or Cloud Storage bucket for further analysis and record keeping. CAI’s enhanced UI makes it even easier for you to build your own flexible queries and quickly get to a comprehensive answer.

Create asset posture visibility 

Cloud Asset Inventory now provides seven types of Asset Insights through the Active Assist platform. These new asset insights help proactively detect anomalies within your organization’s IAM policies, which may be opportunities to improve your security posture. The insights can be aggregated at the Organization, Folder or Project level. 

The seven new Asset Insights include: 

• External members in IAM policies. 
• External users that impersonate your service accounts.
• External members as policy editors.
• External users who can view cloud storage buckets.
• Terminated users/groups that are still in IAM policies
• IAM policies containing all users or all authenticated users.
• Projects with only terminated users as owners.

As a Google Cloud customer you can get started and use all the recently released capabilities and features immediately; check out our documentation to see how. We’d love to hear your feedback; email us with any questions or concerns!

Cloud Firestore is a NoSQL document database that simplifies storing, syncing, and querying data for your mobile and web apps at global scale.

Cloud Firestore is a fast, fully managed, serverless, cloud-native NoSQL document database that simplifies storing, syncing, and querying data for your mobile, web, and IoT apps at global scale.

Its client libraries provide live synchronization and offline support, while its security features and integrations with Firebase and Google Cloud Platform (GCP) accelerate building truly serverless apps.

Here’s other stuff it’s good at:

Sync data across devices, on or offline

With Cloud Firestore, your applications can be updated in near real time when data on the back end changes. This is not only great for building collaborative multi-user mobile applications, but also means you can keep your data in sync with individual users who might want to use your app from multiple devices.

With Firebase Realtime Database, we felt we had built the best force-plate testing software on the market. Thanks to Cloud Firestore, in only two weeks, we built a system that’s significantly better and includes features we never thought possible to ship on Day 1.

Chris Wales, CTO, Hawkin Dynamics

Cloud Firestore has full offline support, so you can access and make changes to your data, and those changes will be synced to the cloud when the client comes back online. Built-in offline support leverages local cache to serve and store data, so your app remains responsive regardless of network latency or internet connectivity.

Simple and effortless

Cloud Firestore’s robust client libraries make it easy for you to update and receive new data while worrying less about establishing network connections or unforeseen race conditions. It can scale effortlessly as your app grows. Cloud Firestore allows you to run sophisticated queries against your data. This gives you more flexibility in the way you structure your data and can often mean that you have to do less filtering on the client, which keeps your network calls and data usage more efficient.

Enterprise-grade, scalable NoSQL

Cloud Firestore is a fast and fully managed NoSQL cloud database. It is built to scale and takes advantage of GCP’s powerful infrastructure, with automatic horizontal scaling in and out, in response to your application’s load. Security access controls for data are built in and enable you to handle data validation via a configuration language.

Over the past few years, many organizations have experienced the benefits of migrating their SAP solutions to Google Cloud. But this migration can do more than reduce IT maintenance costs and make data more secure. By leveraging BigQuery, SAP customers can complement their SAP investments and gain fresh insights by consolidating enterprise data and easily extending it with powerful datasets and machine learning from Google. 

BigQuery is a leading cloud data warehouse, fully managed and serverless, and allows for massive scale, supporting petabyte-scale queries at super-fast speeds. It can easily combine SAP data with additional data sources, such as Google Analytics or Salesforce, and its built-in machine learning lets users operationalize machine learning models using standard SQL — all at a comparatively low cost. 

If your SAP-powered organization is looking to supercharge its analytics with the strength of BigQuery, read on for considerations and recommendations for modeling with SAP data. These guidelines are based on our real-world implementation experience with customers and can serve as a roadmap to the analytics capabilities your business needs.

Considerations for data replication 

Like most technology journeys, this one should start with a business objective. Keeping your intended business value and goals in mind is critical to making the right decisions in the early steps of the design process.

When it comes to replicating the data from an SAP system into BigQuery, there are multiple ways to do it successfully. Decide which method will work best for your organization by answering these questions:

• Does your business need real-time data? Will you need to time travel into past data?
• Which external datasets will you need to join with the replicated data?
• Are the source structures or business logic likely to change? Will you be migrating the SAP source systems any time soon? For instance, will you be moving from SAP ECC to SAP S/4HANA?

You’ll also need to determine whether replication should be done on a table-by-table basis or whether your team can source from pre-built logic. This decision, along with other considerations such as licensing, will influence which replication tool you should use.

Replicating on a table-by-table basis
Replicating tables, especially standard tables in their raw form, allows sources to be reused and ensures more stability of the source structure and functional output. For example, the SAP table for sales order headers (VBAK) is very unlikely to change its structure across different versions of SAP, and the logic that writes to it is also unlikely to change in a way that affects a replicated table. 

Something else to consider: Reconciliation between the source system and the landing table in BigQuery is linear when comparing raw tables, which helps avoid issues in consolidation exercises during critical business processes, such as period-end closing. Since replicated tables aren’t aggregated or subject to process-specific data transformation, the same replicated columns can be reused in different BigQuery views. You can, for instance, replicate the MARA table (the material master) once and use it in as many models as needed. 

Replicating pre-built logic
If you replicate pre-built models, such as those from SAP extractors or CDS views, you don’t need to build the logic in BigQuery, since you’re using existing logic. Some of these extraction objects have embedded delta mechanisms, which may complement a replication tool that can’t handle deltas. This will save initial development time, but it can also lead to challenges if you create new columns, or if customizations or upgrades change the logic behind the extraction. 

It’s also important to note that different extraction processes may transform and load the same source columns multiple times, which creates redundancy in BigQuery and can lead to higher maintenance needs and costs. However, replicating pre-built models may still be a good choice, since doing so can be especially useful for logic that tends to be immutable, such as flattening a hierarchy, or logic that is highly complex.

How you approach replication will also depend on your long-term plans and other key factors — for example, the availability (and curiosity) of your developers, and the time or effort they can put into applying their SQL knowledge to a new data warehouse. 

With either replication approach, bear in mind when designing your replication process that BigQuery is meant to be an append-always database — so post-processing of data and changes will be required in both cases. 

Processing data changes

The replication tool you choose will also determine how data changes are captured (known as CDC – change data capture). If the replication tool allows for it (for example as SAP SLT does) the same patterns described in the CDC with BigQuery documentation also apply to SAP data. 

Because some data, like transactions, are known to be less static than others (e.g., master data), you need to decide what should be scanned in real time, what will require immediate consistency, and what can be processed in batches to manage costs. This decision will be based on the reporting needs from the business.

Consider the SAP table BUT000, containing our example master data for business partners, where we have replicated changes from an SAP ERP system:

In an append-always replication in BigQuery, all updates are received as new records. For example, deleting a record in the source will be represented as a new record in BigQuery with a deletion flag. This applies to whether the records are coming from raw tables like BUT000 itself or pre-aggregated data, as from a BW extractor or a CDS view.

Let’s take a closer look at data coming particularly from the partners “LUCIA” and “RIZ”. The operation flag tells us whether the new record in BigQuery is an insert (I), update (U) or deletion (D), while the timestamps help us identify the latest version of our business partner.

If we want to find the latest updated record for the partners LUCIA and RIZ, this is what the query would look like:

  SELECT partner,
        ARRAY_AGG(i1 ORDER BY i1.recordstamp DESC LIMIT 1) AS row
FROM SAP_ECC.but000 i1 
WHERE partner in ('LUCIA','RIZ')
    GROUP BY partner

With the following result:

After identifying stale records for “LUCIA” and “RIZ” business partners, we can proceed to deleting all stale records for “LUCIA” if we do not want to retain the history. In this example, we are using a different table to which the same replication has been done, for the purpose of comparison and to check that all stale records have been deleted for the selection made and that we only kept last updated records. For example:

  DELETE SAP_HANA.but000 i1
WHERE
i1.recordstamp < TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 3 HOUR) AND
i1.recordstamp < (SELECT MAX(recordstamp) FROM SAP_HANA.but000 i2
WHERE
i1.partner = i2.partner
and partner="LUCIA")

You can also use the following query to retrieve stale records for “LUCIA” partner before moving forward with deletion

  SELECT partner, operation_flag, recordstamp  FROM SAP_HANA.but000 i1
WHERE
i1.recordstamp < TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 3 HOUR) 
AND
i1.recordstamp < (SELECT MAX(recordstamp) FROM SAP_HANA.but000 i2 
WHERE 
i1.partner = i2.partner
and partner="LUCIA")

Which produces all of the records, except the latest update:

Partitioning and clustering

To limit the number of records scanned in a query, save on cost and achieve the best performance possible, you’ll need to take two important steps: determine partitions and create clusters. 

Partitioning
A partitioned table is one that’s divided into segments, called partitions, which make it easier to manage and query your data. Dividing a large table into smaller partitions improves query performance and controls costs because it reduces the number of bytes read by a query.

You can partition BigQuery tables by:

• Time-unit column: Tables are partitioned based on a “timestamp,” “date,” or “datetime” column in the table.
• Ingestion time: Tables are partitioned based on the timestamp recorded when BigQuery ingested the data.
• Integer range: Tables are partitioned based on an integer column.

Partitions are enabled when the table is created, as in the example below.  A great tip is to always include the partition filter as shown on the left-hand side of the query.

Clustering
Clustering can be created on top of partitioned tables by applying the fields that are likely to be used for filtering. When you create a clustered table in BigQuery, the table data is automatically organized based on the contents of one or more of the columns in the table’s schema. The columns you specify are then used to colocate related data.

Clustering can improve the performance of certain query types — for example, queries that use filter clauses or that aggregate data. It makes a lot of sense to use them for large tables such as ACDOCA, the table for accounting documents in SAP S/4HANA. In this case, the timestamp could be used for partitioning, and common filtering fields such as the ledger, company code, and fiscal year could be used to define the clusters.

A great feature is that BigQuery will also periodically recluster the data automatically.

Materialized views

In BigQuery, materialized views are precomputed views that periodically cache the results of a query for better performance and efficiency. BigQuery uses precomputed results from materialized views and, whenever possible, reads only the delta changes from the base table to compute up-to-date results quickly. Materialized views can be queried directly or can be used by the BigQuery optimizer to process queries to the base table.

Queries that use materialized views are generally completed faster and consume fewer resources than queries that retrieve the same data only from the base table. If workload performance is an issue, materialized views can significantly improve the performance of workloads that have common and repeated queries. While materialized views currently only support single tables, they are very useful common and frequent aggregations like stock levels or order fulfillment.

Further tips on performance optimization while creating select statements can be found in the documentation for optimizing query computation.

Deployment pipeline and security

For most of the work you’ll do in BigQuery, you’ll normally have at least two delivery pipelines running — one for the actual objects in BigQuery and the other to keep the data staging, transforming, and updated as intended within the change-data-capture flows. Note that you can use most existing tools for your Continuous Integration / Continuous Deployment (CI/CD) pipeline — one of the benefits of using an open system like BigQuery. But, if your organization is new to CI/CD pipelines, this is a great opportunity to gradually gain experience. A good place to start is to read our guide for setting up a CI/CD pipeline for your data-processing workflow.  

When it comes to access and security, most end-users will only have access to the final version of the BigQuery views. While row and column-level security can be applied, as in the SAP source system, separation of concerns can be taken to the next level by splitting your data across different Google Cloud projects and BigQuery datasets. While it’s easy to replicate data and structures across your datasets, it’s a good idea to define the requirements and naming conventions early in the design process so you set it up properly from the start. 

Start driving faster and more insightful analytics

The best piece of advice we can give you is this: Try it yourself. Anyone with SQL knowledge can get started using the free BigQuery tier. New customers get $300 in free credits to spend on Google Cloud during the first 90 days. All customers get 10 GB storage and up to 1 TB queries/month, completely free of charge. In addition to discovering the massive processing capabilities, embedded machine learning, multiple integration tools, and cost benefits, you’ll soon discover how BigQuery can simplify your analytics tasks. 

If you need additional assistance, our Google Cloud Professional Services Organization (PSO) and Customer Engineers will be happy to help show you the best path forward for your organization. For anything else, contact us at cloud.google.com/contact.

Founded in 2006, PaGaLGuY started as a forum that enabled students, typically aged between 20 and 30 to discuss and seek advice on academic issues. By 2011, PaGaLGuY had increased its traffic to about 250,000 page views per month. The business is now one of India’s largest education networks and provides an app that users can download to their Android and iOS devices.

Over the past six years, PaGaLGuY has extended its service to include video advice from experts on education topics and grown the number of views of its pages to 1.5 million per day. As Head of Technology for the business, Sandeep Kalidindi has played a key role in ensuring PaGaLGuY is as engaging as possible to users. “Because the product is advertising based, the greater the user engagement, the greater the advertising revenue,” Kalidindi explains.

Google Cloud Platform Results

• Supported growth to 1.5 million page views per day and demand spikes that see requests increase from about 90 per second to about 1,200 per second
• Reduced API latency from about 1 second to about 40 milliseconds
• Reduced system administration time from three to four days per week to 30 minutes every two weeks

In 2015, PaGaLGuY’s senior management team decided to deliver an even more relevant experience for users of the education network. “The core thing we had to do was personalise the experience for each and every student that visited PaGaLGuY,” Kalidindi says. “So we had to capture each student’s data to customise what they see when they open the site.”

The business also found traffic to the network was straining its infrastructure. During demand peaks, created by exams involving as many as 5 million students, PaGaLGuY would be inaccessible for periods of 30 minutes to one hour. Furthermore, average API latency had climbed to an unacceptable 1 second, compromising performance.

PaGaLGuY needed to access extensive compute resources to undertake its planned change. Had the business relied on a physical technology architecture to undertake the transformation, it would have had to purchase capacity equivalent to 16 new servers. “There was no way with a small team we could grow to that extent in a short time,” Kalidindi says. “This was the right moment for us to explore cloud services.”

The business established two primary requirements the selected cloud service needed to meet. First, PaGaLGuY had to be able to scale the platform with costs rising only in proportion to the increase in resources consumed. Accordingly, the business would have to minimise the number of employees required to manage the cloud environment. Second, the platform had to give PaGaLGuY easy access to student data and the ability to undertake prompt, granular analysis.

PaGaLGuY reviewed available public cloud services and determined that Google Cloud Platform (GCP) was the best fit for its business. “Google Cloud Platform was considerably more mature than the alternatives, with a high degree of automation and a suite of managed services,” Kalidindi says. PaGaLGuY management then discussed with Google how to optimise cost, performance and availability of its personalised education network on GCP.

With assistance from Google and business transformation specialists Searce, PaGaLGuY was able to deliver the platform into production on GCP in 10 months. “Searce was very proactive in ensuring the environment met our needs and allowing us to gain priority access to Google services in development,” Kalidindi says. “Their team was integral to the success of the migration.”

PaGaLGuY has been running in production in GCP for two years. The education network’s GCP architecture comprises a scalable back-end built on Google App Engine; a managed environment for its containerised applications in Google Kubernetes Engine; messaging-oriented middleware through Google Cloud Pub/Sub; a relational database in Google Cloud SQL; a managed data analytics warehouse running in Google BigQuery; stream and batch data processing through Google Cloud Dataflow; and object storage in Google Cloud Storage.

PaGaLGuY has leveraged GCP services to break down its platform application from a monolithic build to a series of microservices running in Google App Engine that enable independent deployment cycles, minimise test and quality assurance overheads and provide clearer monitoring and logging.

Running on GCP has enabled PaGaLGuY to add new personalisation features and grow fourfold without having to add any new engineers or administrators to accommodate the increased traffic. The business has also used the platform to seamlessly collect and aggregate students’ data for analysis, reporting and delivering a more targeted user experience. Furthermore, PaGaLGuY has been able to provide its management team with direct access to Google BigQuery to scrutinise data rather than require them to wait at least a day to view reports created by the product or technology teams.

Support demand peaks of 1,200 requests per second

“Thanks to Google Cloud Platform, we can easily support demand peaks that see requests per second rise from an average 90 per second to about 1,200 per second for as long as 45 minutes,” Kalidindi says. Due to GCP’s scalability, PaGaLGuY can ensure its education network remains available and performance remains consistent during those periods.

Latency cut to 40 milliseconds

The business has also reduced average API latency from 1 second to about 40 milliseconds. Furthermore, using GCP has enabled PaGaLGuY to automate most of its processes and reduce system administration requirements from three to four days a week across its team members to about half an hour per week.

The performance of GCP has transformed PaGaLGuY’s culture and processes. “Once our team was exposed to Google Cloud Platform and understood the superiority of the platform, our mindset changed from ‘let us do everything on our own’ to ‘let us do what we do best’ and delegate the remainder,” Kalidindi says. The quality of the service provided by GCP means PaGaLGuY effectively considers the cloud provider as part of its team. “We are always eager to see what new services are being launched and are extremely excited about what Google Cloud Platform can provide as part of its roadmap.” he concludes.

As I was preparing for my fireside chat with Dmitri Alperovitch, Founder and Former CTO of CrowdStrike and Executive Chairman of the Silverado Policy Accelerator, for the Google Cloud Government Security Summit taking place on Tuesday, July 20th, I was reflecting about recent developments in cybersecurity, Zero Trust best practices – including aspects of Google’s journey – and the benefits that this approach to cybersecurity brings to the public sector. 

As Dmitri and I discussed what we wanted to share during our session at the Government Security Summit, I quickly realized how pertinent the information was to our public sector audience. For that reason, ahead of Tuesday’s event, I wanted to give you a glimpse of what is to come in our fireside chat. Read below for an exclusive preview of what we plan to discuss. If you are planning to attend on Tuesday, we encourage you to ask questions through our interactive chat function and sign up for 1:1 time with our Ask the Expert program. 

Check out my chat with Dmitri below…

Heather: What do you see as the most significant changes in both the attack surface and cybercriminal tactics in recent years? And why do these changes present unprecedented challenges for public sector? 

Dmitri: In terms of the threat landscape, the more things change, the more they stay the same. The adversaries haven’t changed that much. It’s actually striking. Not all threats are emanating from governments or state actors – either directly or by providing safe harbor to criminals – but a large portion are. That has not changed. The volume and scale of attacks have grown, however, and become massive. And now, the greatest challenge is that everyone is a target. Before, only top-end institutions and government organizations were facing nation state threats. That has changed either through targeted attacks or accidental ones, such as WannaCry. The question now is “How do we protect the most vulnerable and extend much-needed protection to smaller organizations that do not have deep security expertise or resources?”

Heather: Earlier this year, the Biden Administration released its Cybersecurity Executive Order (EO). What do you see as the most important takeaways and potential impact of the EO? 

Dmitri: The Cybersecurity EO is set to have a significant impact. The biggest change is a shift in strategy. For many decades, we all had the mentality that we have to keep attackers out of the network. That is great in theory, but we all know that it is a virtual impossibility with a network of any significant size. Adversaries will find a way in. And, we used to think it was game over once they were in the network.

Then about 10 years ago, we began to see a change. Google led the way with its BeyondCorp strategy as did Lockheed Martin with its sentinel Kill Chain framework. Now, when an adversary penetrates a network, it’s not the end of the game, but the beginning. That is the case if the organization is prepared with a network architecture built around Zero Trust principles. The adversary now has to move laterally, steal credentials, and elevate privileges to get to the resources they want. A Zero Trust architecture can slow them down and give organizations time to detect and eject adversaries from the network. The EO recognizes the power of this approach and has made Zero Trust its fundamental tenet.

Heather: Federal agencies have spent the last 60 days looking at Zero Trust and developing their plans. What do you hope to see? What is critical to accelerating implementation?

Dmitri: The EO establishes a very ambitious timeline, and the US government is not known to move rapidly due to a number of restrictions and considerations. That said, the EO recognizes the idea that logging has to be at the center of modern architecture. Agencies need full visibility into what is going on at endpoints and across the network, have to hunt continuously across networks for adversaries, and must work to rapidly eject them. That is at the core of the EO, and it is a great shift. The private sector should be watching and learning as well.

Heather: What pitfalls should the Federal government look out for in moving forward to implement the EO?

Dmitri: One of the main challenges is the limited authority of the President. Congress is the only branch with the power to change the laws. While the establishment of the Cybersecurity and Infrastructure Security Agency (CISA) a few years ago allows for hunting across agency networks, CISA still has limited authority. It cannot manage the cybersecurity of more than 120 civilian agencies.

Some agencies are performing well when it comes to security, but we have to recognize that this is not the norm. Not all agencies have the capacity to recruit the best cybersecurity talent, but their needs may be just as great. We need to centralize capabilities and leverage the cloud, like Google, to provide all agencies with the best cyber talent and resources.

Join us at the Google Cloud Government Security Summit to hear the rest of our conversation. The event is complimentary. Register today to reserve your spot on July 20th. If you have any questions about the event, please reach out to us at cloudsummitsupport@google.com.