admin – Data Engineering with Salim Saeedi

Databricks not recognizing updated Service Principal Secret

February 4, 2023August 31, 2022 by admin

Problem One way of authenticating Databricks APIs to Azure platform is using a service principal with permissions on Azure AD. In case a client secret is chosen for authentication, Azure forces an expiry date on the secret. When the secret expires the Databricks loses visibility to Azure with an error like below. This error can … Read more

Inline Try-Catch for Azure Data Factory activity output

February 4, 2023March 1, 2021 by admin

At the moment Azure Data Factory expressions and functions are limited and there is not try-catch if you need to make the code to behave differently in case an activity’s output was success or fail. Here is a workaround: Suppose you have a lookup and you need to set the variable based on the output … Read more

Using MERGE for SCD Type 2 with multiple updates

February 4, 2023December 24, 2020 by admin

Below is the typical MERGE statement for SCD Type 2: Since a MERGE statement cannot update more than one match at a time, in scenarios which the source contains historical data that for each matching key (per ON statament) may need more than 1 action in a single merge statement run (e.g. one INSERT and … Read more

Dynamic 365 Audit History Export through Power BI

February 4, 2023December 18, 2020 by admin

Dynamic 365 provides an Audit History in all pages. In case you need to export these data or have a query to the data, Dynamics API provides this access. In Power BI, OData Feed enables this access. In order to achieve this: Use OData source and point it to “audits” entity. Refer to Microsoft’s Web … Read more

Pipenv dependency resolution procedure

February 4, 2023May 9, 2020 by admin

Problem: You are trying to install or uninstall a python package via pipenv and receive: “Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.”