Influence Looker’s Roadmap and join the BI Customer Council (BICC)
Would you like to influence Google Cloud’s BI roadmap more? I’d like to invite you to join the Google Cloud BI...
Data Analytics
Want more from your data? Have questions about products like BigQuery, Dataflow, or Pub/Sub? You're in the right place. Join conversations around data warehouse modernization, streaming analytics, business intelligence, and more.
Focused on Looker? Engage with the Looker Community here.
Focused on Looker? Engage with the Looker Community here.
Forum Posts
Unable to trigger cloud function based on dataflow success event
I am using the google provided template 'text on gcs bucket to bigquery' to move some ndjson files from cloud ...
Best option to synchronize DAGs in Airflow
Hello,I'm new to Airflow and am creating some DAGs that will be chained (serialized) using the TriggerDagRunOp...
Billing bytes changes if insert into table
I have a select that, alone, is billing 1.61 GBHowever, if I insert the data into a table, it jumps to 22.65GB...
Publish Data Quality Scan to Dataplex Catalog UI with Python
When I create a data scan in python, I want to publish it to the Dataplex Catalog UI interface at the same tim...
GA4 backfill - data transfer service
Hello,Like many GA4-BigQuery users I've misunderstood the GA4 data "backfilling" capabilities. This is partial...
Datastream latency issue
We are facing 80 seconds of latency in Datastream Mysql to Bigquery.Reducing the binlog size configuration wil...
Local testing for cloud composer
Hello, Currently, we use cloud composer to use apache airflow (1.10.14+composer).Before we deploy our code, we...
Specify schema for duplicate table dependencies.
Hello, apologies for any error, this is my first question.I have a table table_overview with a dependency to t...
Split table to smaller tables base on column values
Hi, I have a table with many columns: A, B, C,...How can I split my table to smaller tables which have same va...
[Feature] Referencing by alias in the same SELECT statement
SELECT 1 + 1 AS test, test * 0.85 AS test2 This is one of the biggest limitations of BigQuery at the moment, w...
Dataform ETLs too slow when run from Airflow
Hello,I'm having two problems when running Dataform ETLs from Airflow. For example, when running an ETL locall...
Ingestion from pubsublite to GCS never works
I'm following the code example herehttps://cloud.google.com/pubsub/lite/docs/samples/pubsublite-spark-streamin...
BQ Table partition data got deleted fully even though 'spark.sql.sources.partitionOverwriteMode' is
We have a date (DD/MM/YYYY) partitioned BQ table. We want to update a specific partition data in 'overwrite' m...
Universal Analytics 360 historical export to BigQuery
https://support.google.com/analytics/answer/3416092?hl=en#zippy=%2Cin-this-articleAccording to this article:Wh...
Datastream vs Data Fusion replication
Hello!Could you explain the difference under the hood between Datastream and Data Fusion using replication job...
Timezone in BigQuery
There are lots of IDs with different timestamp and time zone information like America/New_York or Asia/Shangha...
Javascript constant is not recognized in my .sqlx file
Hello all,I have been trying to figure out this error message below. I am following the Dataform documentation...
pubsub and key rotation
Credentials to access google pubsub is rotated every month and there is an overlap of 10 days between old and ...
Conditionally trigger files/flow in Dataform based on a login
Hello Team,Is there a way for us to implement a if / else kind of mechanism in Dataform.For Example : If an as...
run a Dataform execution in a specific region (US vs EU)
Hi,I'm new to Dataform, but have previously used dbtin dataform (cloud), I have a repo named "mydata"inside, I...
Subscriber’s activity not reflecting under Usage metrics Analytics Hub
HiI’ve created a listing under my exchange and added one of my friend as subscriber, but still the activity th...
DELETE/UPDATE/MERGE DMLs are currently not supported over tables that had UPSERT streams.
Hi , i am working on migrating data from rds to bq using datastream. and after the load has been completed i a...
Spring based Java API/Library for GCP Bigquery Storage Write API
Hi @ ,Earlier I was using Bigquery InsertAll() method for streaming data to BQ using Spring based library/apis...
BigQuery Data Transfer - Permission denied when transferring public dataset
I am trying to transfer a public dataset to our project. I am specifying a Scheduled Query with a service acco...
Pre-Commit Hook while utilizing Dataform GUI
We are attempting to implement more standardized SQL Formatting across our Dataform repositories. Most of our ...
null values in export data bigquery
I'm trying to export tables from BigQuery into a bucket in google cloud storage in JSON format. it seems that ...
Unable to Create customer expression for date
Hello All,I am trying to create the following customer expression in Looker and getting the error expression i...
Data missing in Bigquery
Hi, We enable both daily export (A full export of data, which takes place once a day) and streaming (Continuou...
Cost Related to sqldump
I need to know is there any cost implications associated with while doing sqldump to export data from mysql cl...
Pivot data in Dataform with EXECUTE IMMEDIATE statements
EDIT: I've figured out how I could do this using operations within Dataform. Article here.Hi everyone,I have a...
Articles from Product Experts and Googlers
Splitting Looker instances: the case for multiple production instances
This article provides a comprehensive guide on how to split productional Looker instance content across multiple instances.
SAP High availability with Windows on Google Cloud Platform : part 2
In part one of this series, we explored how to achieve high availability for the SAP application layer using Windows Server Failover Clustering (WSFC) on Google Cloud Platform (GCP). Now, let's dive into the crucial aspect of database layer high availability, focusing on MS SQL Server's Always On Availability Groups.
SAP High availability with Windows on Google Cloud Platform : part 1
This two-part series focuses on achieving high availability for your SAP deployments on Google Cloud Platform (GCP) using Windows Server Failover Clustering (WSFC). In part one, we'll delve into the intricacies of ensuring SAP application layer high availability.
