So, if the ADF deals with large datasets, then you may avoid group by while fetching the data and use Aggregate transform along with select transform to remove duplicate data. If we group the data for multiple columns, then it would be an extra time-consuming activity for the database server. Distinct Employees Outputįinally, as we know, that group by operation is a CPU intensive operation, especially for large datasets in any T-SQL language. To validate the pipeline execution output, go to Storage Account and navigate to container and see the output file.

Now, pipeline execution will start and complete in some time. To run the pipeline, select the pipeline and click on the debug option. So, the next step is to create a pipeline, drag and drop Data flow activity and associate it with the data flow and make sure to publish the changes. Click on “+” sign to add transforms.ĭata flow Diagram RemoveDuplicateDataflowĬlick on the image to enlarge it transform settings.Īfter performing all the steps have shown above, the data flow is ready now. Data flow requires a Source, Aggregate, Select and Sink transform, and required settings are as shown for each transformation. To create data a mapping data flow, Go to Factory Resources > Data Flows > New mapping data Flow. Now, we are all set to create a mapping data flow. The steps to create services and components have not been in detail, but an Azure data factory implementation is here with a copy activity. This data set would be used in Sink Transformation.

Create a blob storage JSON dataset using the linked service in the ADF.

Create a blob storage JSON dataset using the linked service, browse the Employee.json, and do the Preview Data in the ADF.

Create a blob storage linked service for the storage account and test the connection in the ADF.

Create an Storage Account and add a container named and upload the Employee.json.

Perform the below steps to set up the environment to implement a data flow. Data flow implementation requires an Azure Data Factory and a Storage Account instance.

Using beatunes to remove duplicates