Module_path = os.path. Module_name, _ = os.path.splitext(dag_file) After reading Data Pipelines with Apache Airflow by Bas Harenslak and Julian de Ruiter I see they are using DAG.test_cycle(), the DAG here is imported from the module but when I run the code I get an error that AttributeError: 'DAG' object has no attribute 'test_cycle'ĭAG_PATH = os.path.join(os.path.dirname(file), “…”, “…”, “dags/**/*.py”)ĭAG_FILES = glob.glob(DAG_PATH, DAG_FILES) At first sight it can be unclear how to test Airflow code. Taking the same concept in this blog post, we can build out a more advanced CI/CD process that pulls pipelines from multiple git repositories, versions the DAGs, verifies successful deployments by running systems tests on the production environment, and includes separate Composer dev and production environments.I am writing a unit test for my ETLs and as a process, I want to test all Dags to make sure that they do not have cycles. Airflow, by nature, is an orchestration framework, not a data processing framework. It emphasizes security by enabling the production environment to be restricted to manual changes.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |