Feb 28dbt YAML validator in JetBrainsdbt and its ecosystem have been growing. We make use of not only the dbt-core but also community tools. For example, dbt labs published a github repository to take advantage of JSON schemas of dbt YAML validator in VisualStudio code. …Dbt4 min readDbt4 min read
Jul 11, 2022Unit testing your dbt packageWe are all seeing the rapid growth of dbt’s popularity today. The number of users and the size of its community is on the rise. There are various aspects of dbt which attract experts in the data world. I love all of the features of dbt too. Specifically, I love…Dbt5 min readDbt5 min read
Jan 27, 2022Find Security Violations of IaC in Private GitHub RepositoryToday, it is getting more common to practice Infrastructure as Code (IaC) to build IT infrastructure as kubernetes manifests and terraform resources. …Infrastructure As Code4 min readInfrastructure As Code4 min read
Dec 23, 2021Automate dbt review on GitHubToday, we are bringing many DevOps concepts to the data engineering world. One of the most basic measures is static code analysis used to flag programming errors, bugs, stylistic errors and suspicious constructs. It makes our programming code more readable and maintainable as well. Especially, integrating static code analysis tools…Dbt5 min readDbt5 min read
Jul 6, 2021Data Status Time Machine on Persisted dbt ArtifactsThe article is brought by Yu as one of the blog post series from Ubie, inc. Ubie automatically generates medical records using an AI-powered patient questionnaire that helps save time and provide better patient care. …Dbt6 min readDbt6 min read
Nov 25, 2020Use Airflow-like macros in dbtI used to use create BigQuery tables with Apache Airflow. These days, I am migrating the queries to dbt, but still use airflow to schedule dbt jobs. One of the obstacles to migrate is Airflow-unique jinja2 macros, such as ds and ts . …Dbt2 min readDbt2 min read
Nov 5, 2020Understanding the scopes of dbt tagsdbt (data build tool) is really a great tool, as I posted “5 reasons why BigQuery users should use dbt” before. Especially, dbt tags is very useful to select models depending on the situation by taking advantage of model selection syntax. In the article, I describe the scopes of dbt…Dbt2 min readDbt2 min read
Oct 2, 2020Reusable CircleCI command to halt if no changed target filesI want to skip unnecessary CircleCI jobs with GitHub, when I change nothing in source code. For instance, consider if we modify only README documentation in a pull request. Do we need to run all unit tests? I don’t think so. Actually, CircleCI provides conditional steps, but it doesn’t work…Continuous Integration2 min readContinuous Integration2 min read
Aug 4, 20205 reasons why BigQuery users should use dbtHow do you implement and test data pipelines with BigQuery to create intermediate tables and manage metadata and data discovery? I used to use Apache Airflow’s operators with BigQuery. However, I basically need to implement code in python and manage the dependencies between BigQuery tables manually. As well as, actually…Bigquery5 min readBigquery5 min read
Jun 16, 2018Introduction to RESTful API with Tensorflow ServingI described how to serve trained tensorflow models with tensorflow serving in Serving Pre-Modeled and Custom Tensorflow Estimator with Tensorflow Serving before. In the article, I explained how to make tensorflow models with estimator and how to serve the models with tensorflow serving and docker. And tensorflow serving starts supporting…TensorFlow4 min readTensorFlow4 min read
